An LLM router is supposed to send each query to the cheapest model that can handle it. Under unified evaluation, many routers — including a commercial one — fail to beat the trivial baseline of always using the single best model. The dominant failure is routing collapse to the expensive model, and the cause is structural: routers are trained to predict scalar quality scores for what is really a discrete ranking decision.
A team adds an LLM router to its stack, and the reasoning is sound. The team’s queries vary in difficulty — some need the strongest, most expensive model, many do not — and paying frontier prices for a query a small model would answer correctly is waste. A router predicts, per query, which model is enough, sends easy queries to cheap models and hard queries to expensive ones, and so holds quality fixed while cutting the bill. The team trains or buys a router, deploys it, watches the cost-per-query line drop, and books the saving.
The line did drop. But “the router reduced cost versus sending everything to the expensive model” is not the comparison that decides whether the router was worth building. The comparison that decides it is the router against the trivial alternative: pick the single model that performs best on the team’s actual traffic, send every query to it, and route nothing. A router earns its place only if it beats that. Many routers do not — and a team that never ran the comparison does not know which kind it has.
This post is about the baseline an LLM router actually has to beat, why a large share of published and commercial routers fail to beat it, why the dominant failure is collapse onto the expensive model, and why the cause is a structural mismatch between how routers are trained and what routing is.
It is tempting to say a bad router “loses to a coin flip,” and the phrase has a certain rhetorical snap, so it is worth saying precisely why it is the wrong baseline and what the right one is.
Random routing — the coin flip — is a weak baseline, and beating it proves little. The baseline that means something is the best fixed policy: identify, on a representative sample of your traffic, the single model with the best cost-quality position, and route every query to it unconditionally. Call it the best-single-model baseline. It requires no router, no prediction, no per-query decision — and it is the thing a router is implicitly claiming to beat, because if a router cannot beat “always use the one best model,” then the router, its training, its serving cost, and its latency are all pure overhead.
LLMRouterBench (arXiv 2601.07206) runs that comparison at scale — a unified benchmark of 23,945 prompts expanded to 391,645 instances across 21 datasets and 33 models — and the headline is the one this post is built on: “several recent approaches, including commercial routers, fail to reliably outperform a simple baseline.” The numbers underneath are sharper than the sentence. The commercial router OpenRouter posted a 24.7% relative regression against the best-single-model baseline — it did not merely fail to beat the baseline, it did meaningfully worse than not routing. And the routers that did help, helped a little: the best gains were “up to a 4% average accuracy gain over the Best Single model,” with a persistent gap of roughly 19% (relative) still separating the best router from oracle routing. The prize is real — the oracle gap shows routing could pay well — but the routers as built are not collecting it, and some are worse than the baseline they were bought to beat.
The failure has a characteristic shape, and it is not random underperformance. It is collapse.
When Routing Collapses: On the Degenerate Convergence of LLM Routers (arXiv 2602.03478) names and dissects it. Routing collapse is the “degenerate convergence to the strongest model as the cost budget increases” — as you give a router more budget to spend, instead of spending it intelligently, it spends it by sending more and more queries to the most expensive model, until it is sending almost all of them there. The paper measures it on RouterBench (arXiv 2403.12031) — the canonical multi-LLM routing benchmark, some 405,000 inference outcomes across eight datasets and eleven models, the standardized cost-and-quality evaluation the other results in this post are also measured against. Under existing routers, “the call rate of the strongest model — GPT-4 — rapidly saturates near 100%,” while an oracle router uses that strongest model for “fewer than 20% of queries.” The oracle proves four-fifths of the queries did not need the expensive model. The real routers send nearly all of them there anyway.
That is the failure restated in operational terms. A collapsed router is an expensive, latency-adding, complicated machine whose behavior is indistinguishable from the one-line policy “always call the strong model” — except the one-line policy is cheaper to run and never mispredicts. The router did not lose a close contest with the baseline. It degenerated into a worse-engineered version of the baseline.
Collapse is not bad luck, and it is not fixed by a bigger training set. When Routing Collapses traces it to a structural mismatch, and the mismatch is the most useful thing in this post because it tells you which routers to distrust on sight.
A router decision is discrete. For a given query, the router must pick one model out of several — it is a ranking, an argmax over candidates. But most routers are not trained to rank. They are trained to predict a scalar score: for each (query, model) pair, a number estimating how well that model would do. At decision time the router computes a score per model and picks the top one. The paper calls this an “objective-decision mismatch”: the training objective is scalar regression, the decision is a discrete comparison, and “small prediction errors can flip relative orderings.”
The mismatch would be survivable if the models were far apart in quality on each query. They are not. The paper reports that on its benchmark, “94.90% of queries exhibit exact ties among candidate models” — for the overwhelming majority of queries, two or more models would produce an equally acceptable answer. When the true gap between models is zero, the router’s decision is decided entirely by the noise in its scalar predictions. And a scalar quality predictor has a systematic bias: the strongest model has the highest expected score, so when predictions are noisy and the real differences are nil, the noise resolves, again and again, in favor of the model with the highest mean. Scale up the budget so the router is allowed to pick the expensive model more often, and it does — on essentially every tie. That is collapse, and it falls out of the training objective. The paper’s fix is to train the router on the decision it actually makes: a ranking objective. Its ranking-based router, EquiRouter, reaches GPT-4-level quality at about 77% of the cost where prior routers needed more than 93% — the gain is the size of the mismatch removed.
The honest other half: the papers that diagnose router failure are not papers arguing routing cannot work. They are papers showing it has been built wrong, and the distinction matters for a team deciding whether to keep its router or abandon the idea.
RouterEval (arXiv 2503.10657), a benchmark assembled from more than 200 million performance records, finds that “most” existing routers leave large gains unrealized relative to the oracle — and also documents a “model-level scaling up” effect, in which a genuinely capable router can surpass the best single model as the candidate pool grows. The headroom is real and it is large. When Routing Collapses is itself a counterexample to defeatism: its EquiRouter beats the baselines convincingly once the objective is fixed. And A Unified Approach to Routing and Cascading for LLMs (arXiv 2410.10347) shows cascade routing improving on RouterBench by 1–4% and on SWE-Bench by 14% — while also identifying the mechanism that limits all of these methods: “quality estimates based on current state-of-the-art methods contain significant noise,” and “lower noise in the quality and cost estimates lead to much better performance.”
So the finding is not “routing is futile.” It is “routers as commonly trained and sold underperform a baseline they should beat, because their objective does not match their decision and their inputs are noisy.” That is a fixable critique, and it is also a buying-and-building warning: the default router is more likely to be overhead than savings.
Three honest qualifications, because the version of this argument that overstates loses the reader who knows the field.
First, “commercial routers fail” rests on a small sample. LLMRouterBench evaluated exactly one commercial router by name; the robustness study How Robust Are Router-LLMs? (arXiv 2504.07113) tested two more, finding them prone to “suboptimal, category-driven decisions” — a BERT-based router, it reports, sends every coding and mathematics query to the most powerful model regardless of difficulty. The pattern is consistent and the mechanism is general, but “commercial routers” is, in the published evidence, a handful of products, not an audited market. State it as the sample it is.
Second, the trivial baseline is “always use the single best model,” not a literal coin flip. Random routing is a weaker bar; this post used the coin-flip phrase only to name the genre of trivial baseline. The result that matters is the specific one — routers failing to beat best-single-model — and it should be quoted that way, not inflated into “worse than random.”
Third, the benchmarks are very recent and move fast. LLMRouterBench and When Routing Collapses are early-2026 preprints; their exact figures may shift, and routing methods built to fix the objective-decision mismatch are arriving. The structural diagnosis — scalar scores for a ranking decision, on near-tied candidates, collapsing toward the expensive model — is the durable part. Treat the percentages as current readings, not constants.
A router is a component with a cost and a failure mode. Evaluate it the way you would evaluate any other.
Benchmark the router against the best single model on your own traffic. The comparison is not router-versus-route-everything-expensive and not router-versus-random. It is router-versus-the-one-best-fixed-model, measured on a representative sample of your real queries. If the router does not beat that, it is overhead — and you have learned that before paying for it in production.
Watch the strong-model call rate as your collapse alarm. Track the fraction of queries the router sends to the most expensive model, and watch how it moves as you change the cost budget. If it climbs toward 100% as the budget rises, the router has collapsed and is now an expensive imitation of “always use the strong model.” That metric belongs on the dashboard next to per-query cost attribution.
Distrust routers trained to predict scalar quality scores. The objective-decision mismatch is the root cause of collapse. Prefer routers trained on the discrete decision they make — ranking or comparison objectives — and ask any vendor directly what their router is trained to optimize.
Evaluate on the cost-quality frontier, not a single operating point. A router can look good at one budget and collapse at another. Sweep the budget, plot cost against quality, and compare the router’s whole curve to the baseline’s, the way eval-driven development treats the full graded comparison as the spec rather than one number.
Read the oracle gap as the prize, not as proof the router works. A large oracle-to-router gap means routing could pay — it does not mean your router is collecting it. Use the gap to decide whether routing is worth more engineering; use the baseline comparison to decide whether to ship the router you have.
Before you keep an LLM router in production:
The team’s cost-per-query line really did drop, and against “send everything to the expensive model” the router really did help. But that was never the contest. The contest was against the single best model for the team’s traffic, chosen once and used for everything — no router, no prediction, no per-query latency — and a router that cannot win that contest is not saving money, it is spending engineering and inference budget to approximate a one-line policy. Run the comparison the router was implicitly making. Watch the strong-model call rate for the collapse. And if you keep a router, keep one trained on the decision it actually makes — a choice among models — not on a score it was never asked to report.
Our taxonomy argued prompt injection is a vulnerability class you contain, not a bug you fix. Here is the quantitative half — against the strongest published defenses, indirect-injection attack success stays high, and for agents that can act it stays alarming.
10 min →
A quality-ascending curriculum beats random shuffling — until a decaying schedule delivers your best data exactly when the learning rate is too small to absorb it. The same coupling flips proxy-model ablations: which dataset wins depends on the schedule, not the data alone.
13 min →
An agent workflow that pauses for human approval and then resumes looks safe — a person clicked approve. But the approval decision and the authority the resumed step runs under are two different objects, and most systems conflate them by carrying the approval as an in-band signal the resumed call trusts. That is a confused-deputy bug: a forged resume, a replayed request, or an approval granted at one gate authorizes an action it was never meant to. The fix is to stop transporting approval and start deriving authority from the trusted approval record at resume time, scoped to the exact suspended step — capabilities enforced at the tool boundary, fail-closed by default.
14 min →