AI Vocabulary · 01 The Model
What is RLHF?
Reinforcement learning from human feedback: people rank a model's answers, and the model is trained to give the kind of answers people prefer.
RLHF is a step of post-training. People compare several answers to the same request and rank them. The model is then trained further with reinforcement learning so that answers people rated higher become more likely.
OpenAI's InstructGPT paper of 2022 made the method famous. A model with 1.3 billion parameters trained this way was preferred over the 175 billion parameter GPT-3, with more truthful and less toxic answers, though it still made simple mistakes.
Why it matters for you
RLHF is a large part of why chatbots sound helpful and polite. It also has side effects: a model trained on human approval can learn to tell people what they like to hear, a pattern known as sycophancy.
Own it
When a model agrees with you too readily, ask it to argue the opposite case or to list weaknesses of your plan. Treat friendliness as style and check the substance.
Where it comes from
The method for language models is described in Training language models to follow instructions with human feedback by Ouyang and colleagues at OpenAI (2022).
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.