AI Vocabulary · 01 The Model
What is Scaling Laws?
The observation that a model's error falls predictably as model size, data and computing power grow.
Scaling laws describe a regularity: make a language model bigger, give it more training data and more computing power, and its prediction error drops along a smooth, predictable curve.
In 2020, Jared Kaplan and colleagues at OpenAI showed this curve held across more than seven orders of magnitude. They concluded that extra budget should go mainly into bigger models.
In 2022, Jordan Hoffmann and colleagues at DeepMind corrected that. After training more than 400 models, they found many large models were undertrained: model size and data should grow at the same rate. Their model Chinchilla had 70 billion parameters, about four times fewer than DeepMind's Gopher, was trained on about four times more data with the same computing budget, and outperformed it.
Why it matters for you
Scaling laws explain why AI companies spend so much on chips, data centres and data, and why running out of good data matters. They are observations, not laws of nature, and they predict error on test text, not usefulness for your task.
Own it
Bigger is not automatically better for you. A small language model with good data can be enough for a narrow task. Measure with evals instead of assuming.
Where it comes from
The term comes from Scaling Laws for Neural Language Models by Jared Kaplan and colleagues (OpenAI, January 2020). The correction is Training Compute-Optimal Large Language Models by Jordan Hoffmann and colleagues (DeepMind, March 2022), often called the Chinchilla paper.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.