AI Vocabulary · 01 The Model
What is Pre-Training?
The first and most expensive phase of training, in which a model learns general patterns from huge, broad data.
In pre-training, a model sees enormous amounts of mostly unlabelled training data and learns by predicting what comes next. The result is a base model. It knows a lot, but it does not yet reliably follow instructions or hold a conversation.
That comes afterwards. Post-training turns the base model into a usable assistant, and fine-tuning can adapt it further to one task.
This two-step pattern, train broadly once and then adapt for many uses, is what defines a foundation model. The Stanford researchers who coined that term in 2021 also warned that the flaws of a base model are passed on to everything built from it.
Why it matters for you
Pre-training decides what a model fundamentally knows, which languages it handles well and where its knowledge cutoff lies. It takes so much data and computing power that only a handful of companies can do it at the frontier.
Own it
You will not pre-train a model, but you choose which one you build on. Ask what data it was pre-trained on, under which licence, and up to which date. With an open-weight model you can take a base model and shape it yourself.
Where it comes from
Training a network on broad data first and then adapting it is older than today's language models. An influential example was BERT by Jacob Devlin and colleagues at Google (2018), which learned from unlabelled text and was then fine-tuned with one extra output layer. The term foundation model comes from On the Opportunities and Risks of Foundation Models by Rishi Bommasani and colleagues at Stanford (2021).
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.