AI Vocabulary · 01 The Model
What is Test-Time Compute?
Computing power spent while answering, so a model can think longer or try several solutions, instead of only during training.
For a long time, better models came mainly from bigger training. Test-time compute is the other lever: spend more computing power at the moment of answering, during inference. The model writes longer chains of reasoning, tries several solutions or checks its own work before it replies. This is how reasoning models work.
When OpenAI introduced o1 in September 2024, it reported that performance rises with more time spent thinking. On a maths competition, AIME 2024, it gave its earlier model GPT-4o 12 percent and o1 74 percent on a single attempt. These are the vendor's own figures.
Independent research points the same way, with conditions. Charlie Snell and colleagues showed in 2024 that spreading the extra compute cleverly per question is more than four times as efficient as simply sampling many answers, and that a smaller model with extra thinking time can beat a model 14 times its size, but only on problems where the small model already has some success.
Why it matters for you
The cost moves from training to every single request: more tokens, more waiting time (latency) and a higher cost per task. That is worth it for hard problems and wasteful for simple ones.
Own it
Use model routing: send hard questions to a thinking model and routine work to a fast, cheap one. Measure which tasks actually improve.
Where it comes from
There is no single coinage. The two key sources are OpenAI's Learning to reason with LLMs (12 September 2024) and Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters by Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar (August 2024).
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.