D5
Cost Lives in the Call Graph, Not the Invoice
- 01Propagate tenant / feature / journey tags through every hop of a multi-call fan-out and sum token usage
- 02Compute cost-per-successful-outcome instead of cost-per-call
- 03Explain token asymmetry, billed reasoning tokens, and prompt-cache pricing without hardcoding vendor numbers
- 04Identify whale tenants in heavy-tailed usage and why they invert margin
One action, many calls
topic 19A single user request rarely maps to a single model call. A planner step, several tool calls, retries, a reflection pass, and per-chunk retrieval calls all fan out from one click. Cost lives in that call graph, not in any one request.
The fix is propagation: attach a tag set — tenant_id, feature, journey_id — and carry it through every hop (OpenTelemetry GenAI conventions plus provider request-metadata), then sum token usage across the whole fan-out.
One user action fans out into planner, tool, retrieval, and retry calls. A tenant tag rides node-to-node; a flat 'cost per call' meter holds steady while a 'cost per task' meter climbs far higher and splits away.
Text description
One user action fans out into planner, tool, retrieval, and retry calls. A tenant tag rides node-to-node; a flat 'cost per call' meter holds steady while a 'cost per task' meter climbs far higher and splits away.
Unit economics: where the money actually hides
Three facts reshape the bill (re-verify current vendor ratios — they drift):
- Token asymmetry — output tokens cost several times more than input tokens.
- Reasoning tokens — hidden “thinking” tokens are still billed, usually at output rates, even though you never see them. (Billing and visibility differ by provider.)
- Prompt caching — cached input is far cheaper than fresh input, with a small write premium — only if your prefix stays token-stable.
Usage is heavy-tailed: a few whale tenants drive most spend and can invert margin while average cost-per-call looks healthy.
Common MisconceptionPer-request cost is the cost of the feature.
CorrectionAn agentic feature fans each request into many chained calls plus reasoning tokens — measure at the task/feature level.
Common MisconceptionInput and output tokens cost the same, and reasoning tokens are free.
CorrectionOutput costs several× input, and hidden reasoning tokens are billed (usually as output).
Production failure modes
- A reasoning model 5–10×'s cost through hidden tokens nobody attributed.
- A runaway agent/tool loop with no per-tenant budget cap.
- A whale tenant inverts margin while blended cost-per-call looks fine.
- Attribution too coarse to find which feature or tenant caused a spend spike.
Retrieval Practice
Check one idea at a time
Your per-call LLM cost is flat, but total spend tripled the month you shipped an agentic feature. Most likely cause?