D1
Continuous Batching and the Two Kinds of Caching
- 01Explain how continuous (iteration-level) batching eliminates idle bubbles and its dependence on dynamic KV allocation
- 02Articulate the throughput vs P99 tail-latency tradeoff and why gains flatten at the compute/KV ceiling
- 03Distinguish prefix (prompt) caching from semantic caching by match type, output guarantee, and correctness risk
- 04Diagnose prefix-order cache busting and semantic-cache false hits
Static vs continuous batching
topic 4Static batching waits for every sequence in a batch to finish before refilling — early finishers leave idle “bubbles” and utilization sags. Continuous (iteration-level / in-flight) batching, introduced by Orca, reschedules after each decoding step: evict finished sequences, admit waiting ones, keep every slot lit. Throughput climbs several-fold (present it as a range, not a fixed multiplier). The lever you trade is throughput vs tail (P99) latency, and gains flatten at the compute/KV ceiling.
As sequences finish, their paged KV blocks free up and are immediately reallocated to newly admitted requests each step — the per-step allocation continuous batching relies on.
Text description
As sequences finish, their paged KV blocks free up and are immediately reallocated to newly admitted requests each step — the per-step allocation continuous batching relies on.
Two kinds of caching
Prefix (prompt) cache vs semantic cache
Prefix caching reuses the computed KV for an exact, token-for-token matching prefix, skips that prefill work, and returns the identical answer — it discounts input/prefill only and carries no correctness risk. Semantic caching embeds the incoming query, vector-matches a prior query→response, and if similarity clears a threshold returns the stored response, skipping the whole call — bigger savings, but a real correctness risk. They are often layered.
| Prefix (prompt) cache | Semantic cache |
|---|---|
| Exact token-prefix match | Fuzzy embedding similarity |
| Returns identical output (safe) | Returns stored response (can be wrong) |
| Skips prefill; discounts input only | Skips the entire call; largest savings |
Common MisconceptionPrompt caching caches the response, so it saves on output tokens too.
CorrectionIt caches the prefix KV and still decodes fresh — the discount is on input/prefill only, never output.
Common MisconceptionSemantic caching is basically free and safe.
CorrectionA loose similarity threshold produces false hits — wrong answers on negations, numbers, and dates — because it never re-runs the model.
Production failure modes
- Prefix-order cache busting: a leading timestamp or request-id invalidates the whole prompt (put static first, dynamic last).
- Semantic-cache false positives from an over-loose threshold.
- Latency cliff when batch size saturates the compute/KV ceiling.
- Monitoring only average latency, hiding a blown P99 tail.
Retrieval Practice
Check one idea at a time
You prepend a per-request session ID to an 8K system prompt and enable prefix (prompt) caching, but you see no cache savings. Why?