AI Vocabulary · 01 The Model
What is Benchmark?
A standard test used to compare models. Useful for orientation, but no substitute for testing on your own tasks.
A benchmark is a fixed set of questions or tasks, such as maths problems, coding challenges or knowledge questions, used to score and rank models. New model releases almost always come with benchmark results.
Benchmarks have limits. They measure narrow skills, they age quickly once models are tuned to them, and test questions sometimes leak into training data, which inflates scores. A model that leads a benchmark is not automatically the best for your work.
Why it matters for you
Use benchmarks to build a shortlist, not to decide. The real question is how a model performs on your documents, your customers and your edge cases.
Own it
Build your own small test set, a few dozen real examples with known good answers, and run candidate models on it. That is your private benchmark, and it is the start of proper evals.
Where it comes from
Benchmark started as a surveyors' word, first recorded in 1838: a fixed mark where a levelling staff could rest, so every later measurement could refer back to the same point. By the 1880s it also meant any standard of comparison. In AI, shared test sets turned that idea into public scoreboards, which is why new models arrive with tables of benchmark results. Etymonline: benchmark · Benchmark in surveying
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.