AI Vocabulary · 01 The Model
What is Post-Training?
Everything that happens after pre-training to turn a raw base model into a helpful, safer assistant.
After pre-training, a base model simply continues text. Post-training teaches it to answer questions, follow instructions and refuse harmful requests.
It usually has two parts. First, people write examples of good answers and the model learns from them. Then people compare several answers, and the model learns from those rankings. This second step is called RLHF, reinforcement learning from human feedback (see reinforcement learning). Today, much of this data is also synthetic.
The effect can be large. In OpenAI's 2022 InstructGPT study, human raters preferred answers from a post-trained model with 1.3 billion parameters over those of the 175 billion parameter GPT-3. That is OpenAI reporting on its own model.
Why it matters for you
Much of what you experience as a model's personality comes from post-training: its tone, what it refuses, how cautious it is, and habits like sycophancy. Two models built on the same base can behave very differently.
Own it
Do not choose by brand. Test how a model behaves on your own tasks with evals. If you want to shape behaviour yourself, that is fine-tuning.
Where it comes from
The idea of learning from human comparisons comes from Deep reinforcement learning from human preferences by Paul Christiano and colleagues at OpenAI and DeepMind (2017). It needed feedback on less than one percent of the system's actions and was tested on games and simulated robots. Long Ouyang and colleagues applied it to language models in Training language models to follow instructions with human feedback (OpenAI, 2022). "Post-training" itself is an umbrella term without a single source.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.