AI Vocabulary · 01 The Model
What is Attention?
The mechanism inside a transformer that lets every word piece weigh how much every other piece matters for its meaning.
Attention is the core of the transformer. For every token in the input, the model calculates how strongly each other token relates to it and mixes their information accordingly. In the sentence ‘The bank raised its rates’, attention lets ‘its’ draw on ‘bank’.
Because all these comparisons can run at the same time, attention suits GPUs and made it possible to train very large language models. The price is that every token is compared with every other one, so long inputs become expensive.
Why it matters for you
Attention explains why a long context window costs more and why models use long inputs unevenly. A Stanford study found that information in the middle of a long input was used markedly less reliably than at the start or the end.
Own it
Put the material that matters most at the start or the end of your prompt, and cut what the model does not need. You pay for every token it has to attend to.
Where it comes from
Attention as the sole building block of a language model was introduced in Attention Is All You Need by Vaswani and colleagues at Google (2017).
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.