D2
Quantization: Fewer Bytes per Weight
- 01Estimate FP16/INT8/INT4/FP8 memory footprints and add KV cache + overhead to VRAM planning
- 02Distinguish weight-only (W4A16) from weight+activation (W8A8/FP8) and which serving regime each accelerates
- 03Contrast GPTQ (Hessian-guided error feedback) with AWQ (activation-salient scaling), both post-training
- 04Predict when quantization hurts (bit-width, model size, outliers, granularity, task) and choose evals that surface it
Counting bytes per weight
topic 6Precision sets how many bytes each parameter costs: FP16 ≈ 2 bytes, INT8 ≈ 1, INT4 ≈ 0.5 (plus group scales, so an effective ~4.25–4.5 bits at group size 128), FP8 ≈ 1. A 70B model is therefore roughly 140 GB in FP16, ~70 GB INT8, ~35 GB INT4 — before you add the KV cache and runtime overhead, which any real VRAM estimate must include.
Weight-only vs weight+activation
W4A16 (weight-only; dequantize to FP16 for the matmul) shrinks bytes moved, speeding memory-bound decode at low batch — but gives little prefill gain. W8A8 / FP8 quantizes activations too, running the matmul on INT8/FP8 tensor cores to accelerate compute-bound prefill and high-batch serving. Same bit-count is not the same behavior: FP8 trades dense-region precision for dynamic range and tolerates outliers better than INT8.
| W4A16 (weight-only) | W8A8 / FP8 (weight+activation) |
|---|---|
| Quantizes weights only; dequant to FP16 for matmul | Quantizes weights and activations; matmul on low-precision cores |
| Speeds memory-bound decode (low batch) | Speeds compute-bound prefill and high batch |
| Little to no prefill speedup | Needs native cores (FP8 on Hopper/Ada/Blackwell) |
GPTQ vs AWQ (both PTQ)
Two cures for the same outlier disease
GPTQ quantizes column-by-column with Hessian-guided reconstruction, pushing each rounding error onto the not-yet-quantized weights. AWQ identifies the few activation-salient channels and protects them via per-channel scaling, then quantizes uniformly. Both use a small calibration set — neither is QAT or retraining. Which wins is setting-dependent.
Common MisconceptionINT4 is just half the quality of INT8.
CorrectionLoss is nonlinear: INT8 is near-lossless, while INT4 — especially on small models — can fall off a cliff.
Common MisconceptionQuantization always makes inference faster.
CorrectionSpeedup is regime- and kernel-dependent: W4A16 helps low-batch decode, not prefill, and FP8 needs native tensor cores.
Accuracy vs bit-width: large models stay flat through 8/4-bit then bend; small models plummet at 4-bit; a flat perplexity line shows the blind spot.
Text description
Accuracy vs bit-width: large models stay flat through 8/4-bit then bend; small models plummet at 4-bit; a flat perplexity line shows the blind spot.
Production failure modes
- Shipping after only a perplexity check — reasoning, math, and code silently regress.
- INT4 on a ~7B model hitting a quality cliff.
- Aggressive KV-cache quantization degrading long-context quality silently.
- Calibration-set mismatch with production traffic.
- Expecting W4A16 to speed up prefill (it won't), or running FP8 without native tensor cores.
Retrieval Practice
Check one idea at a time
What distinguishes AWQ from GPTQ?