AI Vocabulary · 06 Own Your AI
What is Distillation?
Training a smaller model to imitate a larger one, so you get much of the quality at a fraction of the cost.
In distillation, a large "teacher" model produces answers to many examples, and a smaller "student" model is trained to reproduce them. The student learns to behave like the teacher on that kind of task while being cheaper and faster to run.
Many small language models are created this way. Companies also use it for their own needs: generate high-quality outputs with a frontier model for one narrow task, then train a small model on them. Note that some providers' terms restrict using their outputs to train competing models, so check the licence.
Why it matters for you
Distillation is how a narrow, repeated task can become cheap to run at scale or small enough to host yourself. It only pays off with high volume and a clearly defined task.
Own it
A distilled model trained for your task, running on your infrastructure, is an asset you control. Treat it like fine-tuning: it needs good data, evals and maintenance.
Where it comes from
In 2006, Cristian Buciluă, Rich Caruana and Alexandru Niculescu-Mizil showed that a small model can learn to imitate a large one, calling it model compression. The memorable name came in 2015 with Distilling the Knowledge in a Neural Network by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, which introduced the teacher and student picture still used today.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.