D2
Knowledge Distillation: Fewer Weights, Forever
- 01Explain distillation as a student matching the teacher's soft logit distribution (dark knowledge), often plus intermediate/sequence matching
- 02Characterize its cost profile: high upfront training, lowest steady-state cost, permanently fewer parameters
- 03Distinguish distillation from fine-tuning a small model on the teacher's generated text
- 04Place distillation among the three techniques and explain why it is lossy with a capability gap
Learning from soft targets
topic 7In distillation a small student is trained to mimic a large teacher’s softened logit distribution — not just the single hard output token. That richer signal, the relative probabilities the teacher assigns to all options, is the dark knowledge. Methods often add intermediate-layer or sequence-level matching on top. The result is a network with permanently fewer parameters.
Common MisconceptionDistillation is just fine-tuning a small model on the teacher's generated text.
CorrectionTrue distillation trains the student to match the teacher's full soft probability distribution, not only its hard output tokens — that distribution carries information plain text labels don't.
Common MisconceptionQuantization and distillation are interchangeable.
CorrectionQuantization cuts precision (bytes per weight); distillation cuts parameter count. They attack different costs and are usually composed, not swapped.
Cost profile and limits
Distillation is expensive to train but yields the lowest steady-state cost, because every inference forever runs a smaller model. It is lossy: a smaller student has a capability gap versus its teacher that surfaces on rare or hard inputs. DistilBERT is the canonical result — about 40% smaller and ~60% faster while retaining roughly 97% of BERT’s language understanding (as cited). Modern examples include Gemma 2 and Llama 3.2’s small 1B/3B prune-plus-distill variants.
| Technique | Defining tradeoff |
|---|---|
| Speculative decoding | Lossless; cuts latency, adds memory, shines at low batch |
| Quantization | Lossy but cheap; cuts memory/bandwidth |
| Distillation | Lossy; high upfront cost, permanently fewer params |
Production failure modes
- A distilled student with capability gaps failing on rare or hard inputs.
- Assuming compression savings from stacked techniques multiply cleanly.
- Treating distillation as a cheap drop-in despite its heavy upfront training cost.
Retrieval Practice
Check one idea at a time
How does knowledge distillation differ from fine-tuning a small model on the teacher's generated text?