Reference

Glossary

The vocabulary of production-LLM engineering.

Filter terms

Agent specificationbehavior contract, agent spec: A testable contract for an agent’s goal, allowed capabilities, state, invariants, exits, and required evidence. It describes behavior independently of a particular prompt or framework.
Checkpointstate snapshot: A persisted snapshot of workflow state that can be loaded after a pause or failure. A checkpoint supports resume and replay, but external side effects still need idempotency.
Context windowtoken window: The finite set of input and output tokens available to one model call. It contains current context, not durable application state or long-term memory.
Conversation engineeringchat engineering, multi-turn context: Designing what carries across turns: roles, selected history, summaries, retrieved facts, reset rules, and escalation. The application creates continuity by supplying state on each new model request.
Decodegeneration phase: The token-by-token generation phase. Each step re-reads weights + KV from HBM for a single-token matrix-vector multiply, so it is bandwidth-bound at low-to-moderate batch and sets per-token latency (TPOT/ITL).
Durable tasktask handle, async task: A long-running operation represented by an explicit identifier that survives disconnection or process restart. Clients use the identifier to poll, provide input, cancel, or fetch the result.
Graph engineeringagent graph, workflow graph: Designing execution as explicit nodes, typed state, and edges. Graphs make branches, parallel work, joins, cycles, waiting, and human gates visible to the runtime and the operator.
GraphRAGgraph RAG: A retrieval approach that extracts entities and relationships, then builds community summaries from a corpus. Optional claim extraction is off by default and needs prompt tuning. GraphRAG helps with relationship-heavy or corpus-wide questions that nearest-chunk semantic search handles poorly.
Idempotencyidempotent tools: A tool call that can be safely retried without changing the result beyond the first application. Essential because agents retry — use idempotency keys for side-effecting actions (payments, sends).
KV cachekey-value cache: The key/value tensors cached per token so attention is not recomputed every decode step. It grows linearly with sequence length × batch and is the dynamic memory bottleneck at serving time — it, not the weights, usually caps concurrency.
Lethal trifecta: Simon Willison’s term for the dangerous combination of private data access + untrusted content + an exfiltration channel. Any two are survivable; all three let injected instructions steal data. Break at least one leg.
LLM-as-judgemodel-graded eval: Using an LLM to grade outputs against a rubric. Scales qualitative evaluation but has biases (position, verbosity, self-preference); calibrate against human labels and use it as one signal, not ground truth.
Long-term memoryagent memory: Information intentionally retained across threads for future work, such as a stable preference or learned fact. It is not the current context window, chat transcript, or workflow checkpoint.
Model Context ProtocolMCP: A standard protocol for connecting an AI host application to external servers that expose tools, resources, and prompts. MCP standardizes the connection; the host still owns consent, permissions, and context.
Multi-agent orchestrationmulti-agent system, agent handoff: Control logic that coordinates multiple specialist agents through explicit routing, handoffs, shared state, or parallel work. It is useful for real isolation or ownership boundaries, not as a default replacement for one well-structured workflow.
PagedAttentionpaged attention: Stores the KV cache in fixed-size, non-contiguous blocks addressed by a per-sequence block table — turning KV memory into a schedulable, low-fragmentation resource and enabling copy-on-write prefix sharing.
Pluginhost plugin: An installable package for a particular host ecosystem. A plugin may bundle tools, skills, or an MCP server, but those concepts are not interchangeable.
Prefillprompt processing: The first inference phase: the whole prompt is processed in one parallel, compute-bound pass. It sets TTFT (time to first token).
Prompt injectionjailbreak via content: Attacker-controlled text in the model’s context overrides intended instructions. Prompt-level defenses are bypassable; enforce permissions in your code and treat model output as untrusted.
QuantizationINT8, INT4, FP8, GPTQ, AWQ: Representing weights/activations in fewer bits to cut memory and bandwidth. Weight-only (e.g. W4A16) vs weight+activation (W8A8/FP8) differ; quality can fall off a cliff at low bit-width, especially for small models — and perplexity hides it, so use task evals.
RAGretrieval-augmented generation: Retrieval-augmented generation: fetch relevant chunks at query time and ground the model’s answer in them. Best for volatile, citable knowledge; quality is gated by retrieval, not the LLM.
Rerankingcross-encoder reranking: A precise second retrieval stage: a cross-encoder jointly scores (query, candidate) pairs to reorder a small candidate set from cheap first-stage retrieval. Too expensive to run over the whole corpus.
Semantic retrievalsemantic search, vector retrieval: Retrieval that embeds a query and candidate text into vectors, then searches for nearby meanings. It handles paraphrase well but should usually be combined with lexical search for exact identifiers and rare terms.
Skillagent skill: A reusable agent capability that may combine instructions, tools, examples, or a specialist workflow. Skill is useful product language, not a universal MCP protocol primitive.
Speculative decodingdraft-verify decoding: A small draft model proposes K tokens that the large target verifies in one parallel pass. With exact verification (vanilla, EAGLE) it is lossless — output is distributionally identical to the target; the draft only affects speed via acceptance rate.