ALLMAcademy

S4

Retry, Fallback, Breakers, and the Degradation Ladder

13 min2 optional deep dives

Separate transient retries from capability-aware fallbacks, circuit breakers, and visible degraded modes.

  1. 01Distinguish retry (same model, transient error) from fallback (different model/provider) and apply each correctly
  2. 02Explain why exponential backoff + jitter and a circuit breaker make retries safe during an outage
  3. 03Identify capability-mismatch failures after a cross-provider fallback
  4. 04Design a degraded-mode ladder that communicates reduced state and uses idempotency keys
Narration — Retry, Fallback, Breakers, and the Degradation Ladder
0:00 / 0:00

Retry ≠ fallback

topic 13
  • Retry the same model on transient errors — 429 (rate limit), provider overload (Anthropic returns 529), standard 5xx, timeouts — using exponential backoff + jitter. (Confirm exact status semantics against current provider docs.)
  • Fallback to a different model/provider only once retries are exhausted or the error is non-transient.
  • A circuit breaker trips open after repeated failures (fail fast, stop hammering), then half-opens to test recovery.

Fallback is not a drop-in

The backup model may not honor the same contract: function calling, strict JSON mode, or vision may be absent or differently enabled. The result is output that goes silently malformed rather than erroring — a capability mismatch, not a transport failure.

Common MisconceptionRetry handles failures, so fallback is unnecessary.

CorrectionRetry only fixes transient errors on the same model; a non-transient or exhausted failure needs a different model — that is fallback.

Common MisconceptionA fallback model is a drop-in replacement.

CorrectionIt may lack the same function-calling/JSON/vision contract, producing silently malformed output — a capability mismatch.

The degradation ladder

Degraded mode is a ladder of rungs, not an error message: full model → smaller/faster → cached/templated → deterministic rule-based → honest failure. Communicate the reduced state (a “fast mode” badge, partial results, disabled features) and use idempotency keys so cross-rung retries don’t double-execute.

Production failure modes

  • Retry storm / thundering herd from immediate, synchronized retries during an outage.
  • Cascading fallback saturating the backup model into its own outage.
  • Capability-mismatch fallback producing silently malformed function-calling output.
  • Breaker stuck open (never recovers) or never tripping (retries into the void).
  • Non-idempotent retry/fallback causing double effects (e.g. duplicate orders).
References & deeper reading (2)

Retrieval Practice

Check one idea at a time

Question 1 of 2

Your primary model returns a 429. What is the correct first response?