ALLMAcademy

Module Gate

Module Gate: Efficiency by Bottleneck

12 min
  1. 01Synthesize speculative decoding, quantization, and distillation by the bottleneck each attacks and its quality risk

All three techniques answer the same fact — at low batch, decode is bottlenecked by memory bandwidth, not FLOPs, because each token streams the whole weight matrix from HBM. They differ in what they move less of and what they cost: speculative decoding (one haul → several tokens, lossless, adds memory), quantization (fewer bytes per weight, lossy, cheap), distillation (fewer weights, lossy, expensive upfront).

Module Gate

Check the whole module

Question 1 of 3

Match each technique (or fact) to its defining tradeoff.

Match each item to its pair.