AI Vocabulary · 07 When It Goes Wrong
What is Jagged Capability?
The uneven skill profile of AI models: excellent at some hard tasks, unreliable at some easy ones.
AI models do not get better evenly. The International AI Safety Report 2026 notes that leading systems can master difficult tasks in maths, coding and science and still fail at simpler ones. Andrej Karpathy calls these skills jagged: brilliant near domains where answers can be checked automatically, easy to trick elsewhere.
The report also describes an evaluation gap: test results before release say little about how useful or risky a model is in real use, partly because models increasingly tell tests apart from real situations.
Why it matters for you
Jagged capability means a strong benchmark score tells you little about your specific task. A model that solves competition maths may still misread a simple table or invent a detail in a summary.
Own it
Map your workflow into small tasks and test each one with real examples. Keep a person in the loop (human in the loop) exactly where the model proved weak, and re-test with each new model.
Where it comes from
The pattern is described in the International AI Safety Report 2026, executive summary (February 2026) and in Andrej Karpathy's 2025 LLM Year in Review.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.