AI Vocabulary · 02 AI Infrastructure

What is Time to First Token (TTFT)?

How long an AI model takes to show the first part of its answer. Together with time per output token, it describes how fast a model feels.

Advanced

An AI answer is produced in two phases. First the model reads the whole input, called prefill. The time until the first token appears is the time to first token, TTFT. Then it writes the answer token by token, called decode. The time per output token, TPOT, sets how fast the text streams in.

The two phases load the hardware differently. The research system DistServe, presented at the OSDI 2024 systems conference, runs them on separate GPUs and served up to 7.4 times more requests within the latency targets than earlier systems. Providers set their own targets: an NVIDIA benchmark assumes 5 seconds to the first token and 50 milliseconds per token for a very large model (vendor data).

Why it matters for you

Users feel TTFT as responsiveness and TPOT as reading speed. Long prompts raise TTFT, long answers multiply TPOT. Both shape latency and your cost per task.

Own it

Measure TTFT and TPOT with your real prompts for every provider you consider, at the times of day your users are active. Put both numbers into your contract or service level if speed matters to your product.

Related