D6
Strategy, Tradeoffs & Silent Regressions
- 01Choose correctly among ICL, RAG, fine-tuning, and distillation by separating the knowledge axis from the behavior axis, and name when each is the wrong tool
- 02Map the inference stack to two latency numbers (TTFT from prefill, per-token from decode) and identify the biggest real cost levers
- 03Reason about the latency/cost/quality/reliability tradeoff surface and explain why no operating point maxes all four
- 04Diagnose silent, confident production failures (stale retrieval, runaway agents, silent eval regressions) and name the matching guardrail
Two axes: what it KNOWS vs how it BEHAVES
topic 22Adaptation methods split on two orthogonal axes, not a single ladder:
- Knowledge, injected at inference (no weight change) — ICL (examples in the prompt) and RAG (retrieve fresh, citable facts).
- Behavior/form, baked into weights — fine-tuning (SFT/LoRA/PEFT) for consistent style, format, or a narrow skill; distillation to compress a validated pipeline into a cheaper student.
Most production systems combine both. The most common mistake is fine-tuning to inject facts.
Common MisconceptionRAG and fine-tuning are competitors — pick one.
CorrectionThey sit on orthogonal axes (knowledge vs behavior) and are usually combined: RAG for fresh facts, fine-tuning for stable form.
Common MisconceptionLatency is one number.
CorrectionIt's two: TTFT from the compute-bound prefill, and per-token (decode) latency that is bandwidth-bound. The biggest cost lever is usually model choice + caching + output-length control, not kernel micro-optimization.
The tradeoff surface
Every inference lever trades among four properties. You optimize a point on the surface; you never max all four.
| Lever | What it buys |
|---|---|
| Large continuous batches | Throughput / lower $-per-token |
| Quantization / distillation | Speed + memory headroom |
| Bigger / reasoning model | Quality |
| More agent autonomy | Capability on open-ended tasks |
Four spokes — latency, cost, quality, reliability. Dragging model-tier or batch-size morphs the achievable polygon; raising quality retracts the latency and cost spokes. A dashed 'ideal' polygon never fills.
Text description
Four spokes — latency, cost, quality, reliability. Dragging model-tier or batch-size morphs the achievable polygon; raising quality retracts the latency and cost spokes. A dashed 'ideal' polygon never fills.
Silent, confident failures
The dangerous production failures throw no errors and pass green infra checks.
Production failure modes
- Stale retrieval: index lag or embedding-version skew → confident wrong answers. Fix: freshness SLAs/TTL, incremental re-index, parity checks.
- Runaway agents: loops, repeated calls, fan-out. Fix: max-step/token/cost caps, no-progress detection, timeouts, circuit breakers.
- Silent eval regression: a prompt edit, dependency bump, or unannounced provider model swap behind a stable alias degrades quality with zero errors. Fix: version prompts + pin model IDs together, golden/regression suites in CI, canary, monitor proxy signals.
- JSON mode is syntax-only: valid JSON can still miss required fields — validate against the schema or use strict structured outputs.
Retrieval Practice
Check one idea at a time
A model fine-tuned on a weekly-changing product catalog keeps returning stale answers. Best fix?