Knowledge Distillation
Training a smaller student model to mimic a larger teacher model, producing a faster cheaper model with most of the teacher's capability.
What Is Distillation
Knowledge distillation is the process of training a small “student” model to reproduce the outputs of a large “teacher” model. The student is much cheaper to run at inference time and typically retains 80–95% of the teacher’s task-specific performance. Use distillation when you have access to a strong but expensive model and need a deployable version that fits on cheaper hardware.
How It Works
- Teacher generates labels — the large model produces output probabilities (or completions) over a training dataset
- Student trains on teacher outputs — instead of training on ground-truth labels, the student learns to match the teacher’s full output distribution (soft labels), which carries more information
- Distillation loss — typically a weighted sum of KL divergence between student and teacher output distributions, plus standard cross-entropy on hard labels when available
- Compute trade-off — distillation is cheap in inference (you only need the student afterward) but expensive in data generation if the teacher is large
Why It Matters
Distillation is how labs ship smaller “mini” or “haiku” variants — Claude Haiku, GPT-5-mini, Gemini Flash — at a fraction of the cost of their flagship sibling. It’s also how open-source teams produce competitive small models without training from scratch: distill from GPT-4 or Claude into a 7B–14B base, fine-tune, ship. The legality of distilling from closed-API models is an open question; most provider terms forbid it.
Examples
- Claude Haiku — distilled from Claude Sonnet/Opus generations
- GPT-5-mini — smaller, faster sibling of GPT-5
- Gemini Flash — Google’s distilled variant
- Orca / Phi — Microsoft’s distilled small models, distilled from GPT-4 outputs
- DistilBERT — the original “distillation works” demonstration in 2019