Most LLM cost discussions end with the same advice: don't send everything to the biggest model. Put a small, fast model in front, escalate when you need to, and watch the bill drop.
That advice is fine. The problem is what it leaves out. The moment you add a router, you've added a second model to your system: one whose job is to make a decision, whose mistakes are mostly invisible, and which almost nobody evaluates with the same care as the models it routes between.
The tutorial version vs. the production version
The tutorial version of a cascade looks like this:
def answer(request):
draft = small_model(request)
if looks_ok(draft):
return draft
return large_model(request)
Three lines of logic, and every one of them hides a decision. What does looks_ok actually check? What happens when it's wrong in either direction? How long did the user wait when the first call was thrown away?
In production, those questions are the system. Here's where it tends to go wrong. (The scenarios below are illustrative, not drawn from a specific deployment.)
Failure mode 1: confident wrong answers never escalate
Escalation usually triggers on things you can detect: malformed JSON, a failed schema check, a refusal, a low self-reported confidence score. What it rarely catches is a well-formed, fluent, wrong answer.
Picture a support assistant that routes "simple" billing questions to the small model. A customer asks about a proration edge case. The small model returns a clean, polite, perfectly structured answer that quotes the wrong number. Every validator passes. The cascade reports a successful cheap resolution. Your dashboard says routing is working.
The failures your router can't see are the ones that cost the most.
Failure mode 2: "difficulty" isn't a feature you can compute
Many routers decide based on a proxy for difficulty: prompt length, keyword lists, a classifier trained on "easy vs. hard" labels someone made up in an afternoon.
But difficulty is relative to the model. A short prompt asking for a regex that handles a nasty edge case is hard. A long prompt asking to summarize a meeting transcript might be easy. If your routing signal is a vague human intuition about difficulty, you're encoding that intuition as a production decision without ever measuring it.
Failure mode 3: the cheap path can be the slow path
A cascade that escalates often pays twice: one call to the small model, then a full call to the large one. If your escalation rate on some task type is high, the "fast first pass" is adding latency, not removing it.
Average latency hides this. Look at end-to-end latency per task type, including retries and validation time, and the picture can flip: for some categories, going straight to the larger model is both faster and simpler.
Failure mode 4: treating all errors as equally expensive
A wrong answer in an internal draft generator costs someone thirty seconds. A wrong answer in a compliance summary or a code change that ships can cost far more. A router optimized purely on token spend will happily save cents on requests where an error costs real money.
What to actually measure
Evaluate the router as its own component, against real traffic, broken down by task type:
per task type:
- quality of the small-model path vs. the large-model path
- escalation rate
- false "ok" rate (validator passed, answer was wrong)
- p50 / p95 end-to-end latency, including retries
- cost per request, and estimated cost of an error
The false "ok" rate is the one most teams don't have, because it requires sampling routed answers and grading them. It's also the one that tells you whether your cheap path is actually cheap.
Where to draw the line
A practical starting point:
- Route to the small model where the output is verifiable (structured extraction, classification against a fixed label set) and errors are cheap or caught downstream.
- Go straight to the large model where errors are expensive, hard to detect automatically, or where the measured escalation rate is high enough that the cascade costs latency.
- Re-measure whenever the models, the prompts, or the traffic mix change. The boundary moves.
Model routing isn't a cost optimization with a classifier attached. It's a decision system, and it deserves the same evaluation discipline as any model you'd put in front of users.
How are you deciding where your cheap path ends? Measured boundaries, or gut feel that hasn't been revisited?
Top comments (10)
The reason model cascades quietly leak money is an implicit symmetric loss function. When teams treat a router as a standard classifier, they usually evaluate it against balanced accuracy or aggregate token savings, as if routing a high-risk edge case to a small model costs the same as sending a trivial string to a large one.
In production, the loss matrix is severely skewed. A false positive costs three cents and an extra round-trip to the frontier model. A false negative that slips past validation produces a silent error that requires manual remediation or customer support triage, which easily costs twenty dollars in labor. The router is not an optimization problem with a single scalar score. It is an exercise in Neyman-Pearson decision theory, where you pin the acceptable tail error rate near zero and let the escalation rate float wherever the empirical distribution demands.
The economic break-even also depends entirely on verification asymmetry. A cascade only creates positive carry when verifying the draft is an order of magnitude cheaper than generating it from scratch. In domains where checking the answer requires re-reading the entire context or writing a secondary semantic grader, the system ends up paying verification latency on every token just to save fractions of a cent on compute.
Exactly, the cost of being wrong isn't remotely symmetric, and token savings alone can hide that pretty well. I'd be careful about assuming verification is cheap, though. If you need another model to judge the answer reliably, you've added cost and latency to every request. At some point, just calling the stronger model upfront is the better deal.
The cutoff is whether the verifier is deterministic or generative.
The math of a cascade only clears when verification cost is nearly zero relative to generation. If verification is structural, such as a schema validator, an AST parser, unit tests, or a regex check, you are paying fractions of a millisecond and zero API cost to catch failure. That leaves plenty of margin to absorb the occasional escalation to the frontier model.
The moment verification requires a second model call to judge semantic correctness, the expected unit cost flips. A cheap generation plus a judge call already burns two round-trips of latency. If the judge rejects even fifteen percent of drafts and escalates, the blended cost of small-plus-judge-plus-frontier exceeds simply calling the frontier model upfront, while doubling your p95 latency. If an output cannot be verified by code, routing directly to the larger tier is almost always the cheaper operational trade.
Yeah, that’s a useful distinction. Structural checks are cheap, but they only tell you what they can actually prove. A valid schema or passing unit tests don't guarantee the answer is correct. I think the real question is how much of the failure space you can cover without adding another expensive step, and whether the remaining risk justifies routing to the stronger model upfront.
The false "ok" rate is the right missing column, and the same grading pass can give you its twin: false escalation. When the cascade throws away a small-model draft and calls the large model, that draft is usually discarded unread. Keep it, and grade a sample of those discarded drafts the same way you grade the accepted ones.
Then each task type gets two error rates from one grading budget: accepted-and-wrong, which is what failure mode 1 costs in quality, and escalated-but-was-fine, which is what failure mode 3 costs in latency and spend. A boundary tuned only against false ok can drift toward escalating everything, and the second number is the only one on your list that shows it happening.
Good point. False escalation is the other half of the picture, and keeping the discarded drafts makes it measurable without needing a separate grading setup. Otherwise, it's easy to convince yourself the router is getting safer when it's really just escalating more than it needs to. Both rates should be part of tuning the boundary.
Agreed — both rates belong on the same tuning pass.
I'd plot them on one boundary sweep: as you move the escalate threshold, track false-ok and false-escalation on the same graded sample of discarded drafts. The operating point you want is where false-ok is under your quality budget and false-escalation hasn't eaten the latency/spend budget. Tuning only against false-ok will walk you toward the right edge of that curve without you noticing until the bill shows up.
Exactly, and I think the interesting part is that the best threshold isn't necessarily the one with the lowest false-ok rate. It's the one that keeps both error rates within acceptable limits without blowing up latency and cost. Otherwise, we're just moving the problem from answer quality to infrastructure spend.
The false ok rate is the metric I'd lean on hardest, and I'd weight its sample toward the task types the router labels simple, since a uniform sample will mostly confirm easy billing questions and rarely hit the proration case. I usually log the router's decision and its input signal next to each answer so a sampled failure can be traced back to whatever sent it down the cheap path.
Agreed, especially on weighting the sample toward requests routed as simple. That's where the blind spots are most likely to hide. Logging the decision and input signals also makes it much easier to understand why a request stayed on the cheap path, rather than just discovering after the fact that the answer was wrong.