D4
Traces, Telemetry, Drift, and the Closed Loop
- 01Read a trace as a span tree and attribute a bad/slow/expensive response to a specific step
- 02Treat LLM-specific signals (output tokens, TTFT/tail latency, schema failures, refusals) as first-class
- 03Distinguish data, concept, and model drift and explain why pinning snapshots matters
- 04Wire the closed loop: traces to sampled failures to golden set to regression gate to release
A trace is a tree of spans
topic 18In OpenTelemetry-style terms, a trace is one request end-to-end; a span is one timed unit — an LLM call, a retrieval, a tool invocation — nesting into a tree. GenAI semantic conventions standardize attributes like model, token counts, and latency. (These conventions were still experimental in early 2026 — verify attribute names against the current spec.)
LLM-specific telemetry
Classic monitoring watches HTTP status, CPU, memory. LLM apps fail — and bleed money — elsewhere:
| Signal | Why it's first-class |
|---|---|
| Output tokens | Pricier than input and dominate latency (autoregressive) |
| TTFT / ITL / tail | p99 can be terrible behind a healthy p50 |
| Semantic errors | Invalid JSON, schema violations, refusals, truncation, tool-call failures — all inside a 200 |
Drift and closing the loop
Data drift (input distribution shifts), concept drift (the right answer changes), and model drift (a silent provider upgrade) all degrade quality quietly. Detect via input/output/score distribution monitoring — PSI, KL, KS — plus online sampled judge evals. PSI thresholds (~0.1 / ~0.25) are a borrowed heuristic, not a law.
Traces surface sampled failures that grow the golden set; the golden set gates the next release through regression checks before deploying back to production.
Text description
Traces surface sampled failures that grow the golden set; the golden set gates the next release through regression checks before deploying back to production.
Common MisconceptionTracing is just structured logging.
CorrectionTraces are causal and hierarchical — a span tree shows which step caused a bad/slow/expensive result; flat logs can't.
Common MisconceptionNo HTTP errors means the system is healthy.
CorrectionLLM failures hide inside 200s: bad JSON, refusals, hallucinations, truncation. Watch semantic signals, not just status codes.
Production failure modes
- Silent provider upgrade: pinned prompt, temp 0, nothing deployed — scores drop overnight.
- Counting unparsed output as success because the HTTP call returned 200.
- A p99 tail pager-storm hiding behind a healthy p50 dashboard.
- Cost blowup from an output-token explosion no one was watching.
- Unredacted PII sitting plaintext in traces.
Retrieval Practice
Check one idea at a time
Your prompt is pinned, temperature is 0, and you deployed nothing — yet eval scores drop overnight. Most likely cause?