D2
Speculative Decoding: Draft, Then Verify
- 01Describe the draft-then-verify loop: a small draft proposes K tokens, the target verifies all K in one parallel pass, accepting the longest correct prefix
- 02Explain why modified rejection sampling makes exact-verification speculative decoding lossless
- 03Predict where speedup collapses: weak draft / low acceptance, high concurrency, or sampling/tokenizer mismatch
- 04Recognize that speculative decoding adds memory and that EAGLE/Medusa variants differ in approach and guarantees
The draft-then-verify loop
topic 5A small draft model proposes K tokens autoregressively (cheap). The large target model then verifies all K in a single parallel forward pass — the same expensive weight load it would have spent on one token now covers many. You accept the longest correct prefix and emit one corrected token at the first divergence, then repeat. Typical speedups are ~2–3×, and more with methods like EAGLE/EAGLE-2/3 or Medusa.
Why it’s lossless
Acceptance uses modified rejection sampling: accept each drafted token with probability min(1, p_target/p_draft), otherwise resample from the normalized positive difference of the two distributions. This provably reconstructs the target’s exact output distribution.
Common MisconceptionSpeculative decoding degrades output quality.
CorrectionExact-verification speculative decoding is lossless — it changes latency via the acceptance rate, not the distribution the model samples from.
Common MisconceptionSpeculative decoding saves memory.
CorrectionIt costs extra memory — you run a draft model (or extra heads) alongside the target; it buys latency, not footprint.
Draft hops out K tokens fast; the target verifies all K at once, accepting up to the first mismatch; an acceptance-rate slider stretches or shrinks the speedup bar.
Text description
Draft hops out K tokens fast; the target verifies all K at once, accepting up to the first mismatch; an acceptance-rate slider stretches or shrinks the speedup bar.
Where the speedup collapses
The whole win rides on the target being memory-bound. At high concurrency/batch the target is already compute-bound, so verifying K tokens in parallel no longer rides a free memory haul. A weak draft with low acceptance means constant rejections — sometimes a net slowdown. And the lossless guarantee assumes the draft and target share the same sampling parameters and tokenizer.
Production failure modes
- Speculative decoding at high concurrency wasting FLOPs for little gain.
- A weak draft → low acceptance → net slowdown.
- Sampling-parameter or tokenizer mismatch breaking the lossless guarantee or acceptance.
- Forgetting it adds memory when planning VRAM budgets.
Retrieval Practice
Check one idea at a time
Why is exact-verification speculative decoding lossless?