Module Gate

Module Gate: Inference Internals & Performance

12 min

01Synthesize prefill/decode, the KV cache, PagedAttention, continuous batching, and the two caching strategies

You’ve seen how one request splits into a compute-bound prefill and a bandwidth-bound decode, why the KV cache (not the weights) caps concurrency, and how PagedAttention, continuous batching, and the two caching strategies all exploit that structure. Lock it in below.

Module Gate

Check the whole module

Question 1 of 3

A single request streams tokens slowly after a fast first token, yet GPU compute utilization is low during streaming. Which phase and bottleneck explains this?

Decode, which is memory-bandwidth-bound (re-reading weights + KV from HBM per token)Prefill, which is memory-bandwidth-boundDecode, which is compute-bound on heavy per-token mathPrefill, which is bandwidth-bound on the KV cache