AI Vocabulary · 01 The Model
What is Inference?
Running a finished model to get an answer. What you pay for every time you send a message.
Inference is the moment a trained model is used: your input goes in, the model generates tokens, an answer comes out. Every message to an AI assistant is an inference request.
Training happens once; inference happens billions of times a day. That is why, for users, inference is the cost that matters. It runs on someone's hardware, either a model provider in the cloud or your own machine with local AI.
Inference cost depends on the model size and the number of tokens in and out. Agents that make many steps also make many inference requests.
Why it matters for you
Your AI running costs are inference costs. They scale with use: more users, longer documents and multi-step agents all multiply them. Plan for that before a pilot becomes a product.
Own it
You can choose where inference happens. Running it locally or on your own infrastructure trades per-token fees for hardware and maintenance, and keeps data in your hands.
Where it comes from
"Inference" is borrowed from statistics, where it means drawing conclusions from data. The split between training and inference became everyday vocabulary during the deep learning boom of the 2010s. NVIDIA's 2016 explainer put it with a school metaphor: training is the education, inference is putting it to work on the job.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.