AI Vocabulary · 01 The Model

What is Gradient Descent?

The method by which a model learns: nudge every parameter a little in the direction that makes its errors smaller, and repeat.

Advanced

During training, a loss function turns each prediction into one number that says how wrong it was. Gradient descent calculates, for every parameter, in which direction that number would fall, and moves all parameters a small step that way. Repeated billions of times, the errors shrink.

The educator Grant Sanderson pictures it as walking downhill on a landscape of errors. The size of each step is the learning rate: too large and the model overshoots, too small and training takes forever. The method usually finds a good valley, with no guarantee that it is the best one.

Why it matters for you

Gradient descent is why training is so compute-hungry: every step touches every parameter of a model with billions of them. It is also why training results vary a little between runs, and why the data you train on shapes everything the model learns.

Own it

If you fine-tune a model yourself, the learning rate and the number of steps are the first settings to test. Start with the provider's defaults and change one thing at a time.

Related