AI Vocabulary · 01 The Model
What is Latency?
The waiting time between sending a request and getting the answer. In AI, speed is a feature.
Latency is how long you wait for an answer. For AI it has two parts: the time until the first words appear, and how fast the rest of the answer streams in, usually measured in tokens per second.
Latency depends on the model's size, how busy the model provider is, how much input you send and how much output is generated. Reasoning models add extra waiting time because they generate working steps first. Small models and shorter inputs are faster.
Why it matters for you
For a chat or a voice assistant, a few seconds of delay feels broken. For an overnight report, nobody cares. Decide per use case how fast it has to be, and pick the model and setup accordingly.
Own it
Speed is part of the user experience you control. Shorter context, smaller models for simple steps and streaming answers often help more than switching to a bigger model.
Where it comes from
Latency comes from the Latin latere, "to lie hidden", the same root as "latent". It first described things concealed, from 1882 the delay between a stimulus and a response, and computing adopted it in the 1950s. A fitting history: latency is the time hidden between your question and the first word of the answer. Etymonline: latency
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.