S4
Model Routing: Cheapest Sufficient Model, Measured Honestly
Route each task to the cheapest model that satisfies capabilities, quality, latency, and risk.
- 01Describe a routing strategy combining hard capability rules, complexity classification, and latency/cost policies
- 02Quantify the order-of-magnitude price gap between frontier and small models and when down-routing pays off
- 03Reason about the router's own added latency and failure as part of the cost equation
- 04Explain why quality regression on down-routed queries is silent and must be measured
Cheapest sufficient model
topic 12Routing sends each request to the cheapest model that still meets the bar, decided by three stacked mechanisms:
- Hard capability rules — needs vision / function calling / a long context → must go to a model that has it.
- Complexity classification — a lightweight classifier (e.g. RouteLLM, NotDiamond, Martian) sorts easy vs hard.
- Latency / cost policies — pick the tier that fits the SLA and budget.
The prize: frontier and small models differ on the order of ~10–30× per token (an order-of-magnitude estimate, not a fixed price), so steering easy traffic cheap compounds fast.
The router has its own cost
A classifier in front of every request adds latency and is itself a failure point. If it adds 200ms and occasionally mis-routes, that overhead and error must net out below the token savings — otherwise routing is a loss disguised as an optimization.
The silent regression
Down-route a hard query and the cheap model rarely errors; it just answers a little worse. Without an explicit quality measurement on down-routed traffic, that degradation never surfaces.
Common MisconceptionRouting always saves money.
CorrectionRouter latency, router failures, and the cost of mis-routed hard queries can erase the per-token savings — and the quality hit is invisible unless measured.
| Aggressive down-routing | Conservative routing |
|---|---|
| Large token-cost savings | Higher spend |
| Higher risk of silent quality loss on hard queries | Lower quality risk, more reliable |
Production failure modes
- Silent quality degradation on hard queries sent to a weak model.
- Router latency + failure overhead exceeding the savings it produces.
- Mis-route of a capability-dependent request (e.g. needs vision) to a model that lacks it.
- No measurement of down-routed accuracy, so the regression is never noticed.
References & deeper reading (1)
- Building Effective AgentsAnthropic Engineering
Retrieval Practice
Check one idea at a time
You add a cost-saving router. Token spend drops, no new errors appear, but users report worse answers on harder tasks. What happened?