AI Vocabulary · 06 Own Your AI
What is Quantization?
Shrinking a model by storing its numbers with less precision, so it runs on smaller, cheaper hardware.
A model's parameters are numbers, normally stored with high precision. Quantization stores them with fewer bits, for example 8 or 4 instead of 16. The model file becomes much smaller, needs less memory and often runs faster.
The trade-off is some loss of quality, usually small at moderate levels and more noticeable at aggressive ones. Quantized versions are what make it possible to run capable open-weight models on a laptop with local AI tools; labels like "Q4" or "4-bit" in model names refer to this.
Why it matters for you
If you want to run AI on your own hardware, quantization decides which models fit at all. It can turn "we need an expensive GPU server" into "it runs on the machines we have".
Own it
Test the quantized model on your real tasks before relying on it. For many business jobs the difference is invisible; for precise reasoning or numbers it can matter. A quick comparison with evals gives you the answer.
Where it comes from
The word comes from signal processing, where continuous signals are rounded to a limited set of levels. For large language models, Tim Dettmers and colleagues showed in 2022 with LLM.int8() that huge models can run in 8-bit precision with almost no loss in quality. Projects like llama.cpp then made 4-bit models a normal way to run AI on laptops.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.