A concrete example
GPT-4 answers. You train a small model to reproduce its answers. The small one becomes almost as good, for 10x less.
Why it matters
Distillation explains why the "mini" models (Haiku, Gemini Flash, GPT-4o mini) are so good despite their small size.
You see it when the goal is to cut costs while keeping quality close.
Don't mix it up with
Fine-tuning: Fine-tuning adapts an existing model on examples to improve one specific behavior.
Quantization: Quantization lowers a model's internal precision to save memory and gain speed.
Common mistakes
- Confusing distillation with fine-tuning: they're not the same thing.
- Believing a distilled model keeps 100% of the teacher model's abilities.
- Forgetting that distillation takes a huge amount of data from the teacher model.
Quick checklist
- First I check whether the word names a concept, a tool, a risk, or a metric.
- I tie it to a concrete case: GPT-4 answers. You train a small model to reproduce its answers. The small one becomes almost as good, for 10x less.
- I keep the main trap in mind: Confusing distillation with fine-tuning: they're not the same thing.
Quick questions
What is Distillation in AI?
Distillation trains a small model to imitate a more powerful one.
Where will I run into Distillation?
You see it when the goal is to cut costs while keeping quality close.
Which word should I read next?
Start with Fine-tuning, Quantization, AI model.