Reinforcement Learning from Human Feedback
A training method that aligns LLM behavior with human preferences using a reward model trained on human-ranked outputs.
What Is RLHF
RLHF is the alignment technique that turned raw pre-trained LLMs into the helpful, mostly-honest assistants you use today. After a model is pre-trained and instruction-tuned, RLHF further adjusts its behavior by training a reward model on human preference data and then optimizing the LLM to produce outputs the reward model scores highly. Use the term when discussing model alignment, why models refuse certain requests, or the difference between a “raw” base model and a “chat” model.
How It Works
- Preference data collection — humans see pairs of model outputs and rank which is better
- Reward model training — a separate model is trained to predict human preferences (output a scalar “this is good” score)
- Policy optimization — the LLM is fine-tuned with reinforcement learning (PPO, DPO, GRPO, or newer variants) to maximize the reward model’s score while staying close to the original distribution
- KL constraint — a divergence penalty prevents the model from “reward hacking” by producing weird outputs that game the reward model
Newer variants — DPO (Direct Preference Optimization), GRPO, RLAIF (using AI feedback instead of human) — skip the explicit reward model and optimize on preference pairs directly.
Why It Matters
Without RLHF, a base LLM is a powerful but unreliable next-token predictor. With RLHF, the model learns to be helpful, refuse harmful requests, follow instructions, admit uncertainty. It’s the difference between GPT-3 (released 2020, fascinating but unusable as a product) and ChatGPT (released 2022, immediately a product). Every major frontier model in 2026 — Claude, GPT, Gemini, Grok — uses RLHF or a successor technique in its final training stage.
Examples
- InstructGPT / ChatGPT — the original RLHF demonstration that made LLMs viable as products
- Claude — uses RLHF + Anthropic’s Constitutional AI (RLAIF variant)
- DPO — simpler successor used by many open-source fine-tunes
- GRPO — DeepSeek’s variant, popularized by R1 reasoning training