Anatomy of an LLM Request

One request, six parts

mental model

1. Messages identify roles and turns.
2. Instructions say what the model should do and what constraints apply.
3. Context supplies the facts, examples, tool descriptions, or retrieved text available now.
4. Tokenization turns that assembled input into token IDs the model can process.
5. Inference and decoding repeatedly estimate and select the next token.
6. Response handling returns text, structured data, or a tool request to the application.

From input to output

Your application first assembles an ordered request. Depending on the API, that request may contain system or developer instructions, user messages, prior turns, tool definitions, images, and retrieved documents. The provider converts the content into tokens.

During inference, the model uses those tokens to estimate a probability distribution for the next token. A decoding strategy selects one candidate. The new token is added to the sequence, and the process repeats until the model or API reaches a stop sequence, an end token, or an output limit.

Controls such as temperature or top-p affect how candidates are selected. They can make outputs more or less varied, but they do not add knowledge, state, or verification.

The response is not the application

The returned response may be prose, JSON-like data, or a request to call a tool. The application still decides what to validate, display, store, execute, retry, or reject.

If the next call needs an earlier fact, the application must do one of three things:

• include it again in the new request,
• retrieve it from an explicit store,
• or use a documented provider feature that stores and reattaches state.

An inference server may use internal caches while generating tokens. That implementation detail is not user memory and does not remove the application’s responsibility for state.

Common request-level mistakes

• Assuming a fact from an earlier call is available without including or retrieving it.
• Treating sampling settings as factuality or correctness controls.
• Executing a returned tool request without validating its name, arguments, and authorization.