ALLMAcademy

S3

The Guarantee Ladder: Structure Is Not Truth

14 min2 optional deep dives

Move from requested formatting to schema-constrained responses, then validate meaning separately.

  1. 01Distinguish the three guarantee rungs (prompt-only / JSON mode / schema-constrained) and name what each does NOT guarantee
  2. 02Explain how constrained decoding works (schema to grammar/FSM to per-token logit masking) and where it forces wrong or empty values
  3. 03Justify treating a typed validator as the gate, since schema validity is not the same as correctness
  4. 04Design a bounded, cheap-first, error-fed repair loop that always degrades to a safe typed default
Narration — The Guarantee Ladder: Structure Is Not Truth
0:00 / 0:00

Three rungs, three different guarantees

topic 8

Reliability is a ladder, and each rung promises something stronger about shape — never about truth.

  • Prompt-only (“reply in JSON”) — guarantees nothing; you may get prose, fences, or invalid JSON.
  • JSON mode — guarantees syntactically valid JSON, not your schema (fields, enums, types can all be wrong).
  • Schema-constrained decoding (OpenAI Structured Outputs strict:true; Anthropic output_config.format with type: json_schema; Google structured output; Outlines / XGrammar / GBNF / lm-format-enforcer locally) — guarantees structure + types.

Anthropic also supports strict: true tool schemas. Forced tool_choice remains useful when the model must call a specific tool, but it is no longer the only route to schema-constrained output.

The Guarantee LadderInteractive · 2D

A JSON object builds token-by-token along an FSM track; at each step a logit bar chart slams grammar-breaking tokens to the floor while legal tokens stay lit. A side rail shows the three rungs and what each does — and does not — guarantee.

Loading diagram…
Text description

A JSON object builds token-by-token along an FSM track; at each step a logit bar chart slams grammar-breaking tokens to the floor while legal tokens stay lit. A side rail shows the three rungs and what each does — and does not — guarantee.

How constrained decoding works (and bites)

Your schema compiles to a grammar/FSM; at each step a logit mask zeroes every token that would break the grammar, so only legal tokens can be sampled. The first complex schema pays a one-time compile latency (then cached).

Common Misconceptionstrict mode guarantees correct data.

CorrectionIt guarantees shape and types only — a required field with no real answer is still filled with a valid, possibly hallucinated value.

Common MisconceptionJSON mode equals Structured Outputs.

CorrectionJSON mode guarantees syntax; Structured Outputs guarantees your schema. Different rungs.

Validation is the gate; repair is cheap-first and bounded

Run a typed validator (Pydantic / zod / jsonschema; Instructor wraps + validates) on every output. Then repair cheapest-first: deterministic json_repair (zero LLM calls) → re-ask with the verbatim validator error at lower temperature → cap at 2–3 retries → terminal safe typed default / partial with low-confidence flag / human review.

Production failure modes

  • Truncation at max_tokens (finish_reason='length') yields a valid prefix but invalid JSON — silently corrupts data if ignored.
  • Enum/type drift and semantically-wrong-but-valid data passing because validation lacked business rules.
  • Unbounded repair loops burning paid round-trips.
  • Schema drift across the cached prompt, the grammar, and the downstream consumer.
References & deeper reading (3)

Retrieval Practice

Check one idea at a time

Question 1 of 2

OpenAI Structured Outputs with strict:true guarantees that the output...