Primer
Context Windows and Token Budgets
Manage the finite space shared by instructions, history, retrieved evidence, tool definitions, and the model's answer.
- 01Budget a context window across input and expected output
- 02Explain how tokenization, truncation, and poor placement can remove or bury useful information
- 03Apply selection, compression, and isolation without confusing context with memory
Budget the whole request
finite resourceA context window is the maximum token span the model can use for one generation. The budget may need to hold:
- instructions and the current user request,
- selected conversation history,
- retrieved evidence and examples,
- tool descriptions and tool results,
- formatting overhead,
- and enough space for the expected output.
Instructions, history, retrieved evidence, and tool definitions fill the input side while an explicit reserve protects space for the response.
Text description
Instructions, history, retrieved evidence, and tool definitions fill the input side while an explicit reserve protects space for the response.
Tokens are not characters
Models process tokens, not words or characters. A token may represent a whole short word, part of a long word, punctuation, whitespace, or a fragment of code. Different tokenizers split the same text differently. Estimate with the tokenizer for the chosen model and leave margin for formatting and output.
When a request exceeds the limit, an API may reject it or some layer may truncate it. Blindly dropping the oldest or middle content can remove an instruction, a decision, or the only relevant piece of evidence.
The hard limit is not the only limit
Long input can fail before it reaches the maximum size:
- Lost in the middle: information buried in a long sequence may be used less reliably than information near clearer boundaries.
- Context rot: irrelevant, stale, duplicated, or conflicting content reduces signal quality as the context grows.
- Instruction competition: old messages or retrieved text can distract from the current task.
More tokens increase available space; they do not guarantee that every token receives equal attention or that conflicting evidence will be resolved correctly.
| Technique | Use it when |
|---|---|
| Select | Only a subset of history, documents, examples, or tools is relevant now. |
| Compress | Older detail can become a faithful summary of decisions, facts, and open work. |
| Isolate | A side task needs substantial context that should not pollute the main task. |
Compression can lose detail, selection can miss evidence, and isolation creates coordination work. Treat each move as a controlled tradeoff and preserve source links or identifiers when a summary may need verification.
Common MisconceptionIf information was in an earlier context window, the model remembers it.
CorrectionOnce it is absent from the current call, it is unavailable unless the application stores and supplies it again.
Context budget failures
- Using the full advertised window for input and leaving too little space for the answer.
- Truncating by position without protecting instructions, recent decisions, and critical evidence.
- Summarizing away identifiers or constraints that later steps must use exactly.
- Giving every tool and document to every step instead of selecting the relevant subset.
References & deeper reading (3)
- Effective Context Engineering for AI AgentsAnthropic Engineering
- Context WindowsAnthropic
- LLM FundamentalsMicrosoft Learn
Retrieval Practice
Check one idea at a time
A model supports a 32K-token context window. Why should an application reserve space before filling the input?