A Leaderboard Is Not a Ranking

LLMRouter does the boring, valuable thing. It formalizes model routing as a sequential decision process with five named parts — context encoders, model encoders, scoring functions, decision rules, learning signals — then ships sixteen-plus reference routers and a benchmark, xRouteBench, spanning 4,767 test queries across generic, memory-augmented, vision, time-series and personalized tracks. The headline is that learned routers beat the strongest fixed-model baseline by 14.6% relative. Ignore the 14.6. The number underneath it that actually changes how you should read every routing result you’ve seen is the cost coefficient β. MLPRouter sits near the bottom of the table under quality-only evaluation and becomes optimal once β ≥ 0.4. The abstract renders this as “lightweight routers become more competitive under tight cost constraints,” which is the diplomatic version. It isn’t a gap narrowing; it’s an ordering inverting. Which means “we evaluated the routers and picked the best one” is not a well-formed claim unless you say what β you used. A router leaderboard with no declared cost axis has taken one arbitrary slice through a surface and published it as a point.

The second thing hiding under the headline matters more for anyone deciding whether to build this at all. The +14.6% does not come from cost arbitrage. It comes from sending queries to smaller models specifically on the queries the largest model handles badly. Those are two entirely different theories of why routing works, and they predict opposite futures. If routing value is a price spread, then frontier inference getting cheap eventually kills routing — you wait it out. If routing value is capability jaggedness, routing survives every price cut and dies only when models get smooth, and nobody has produced evidence that they are getting smoother. Most routing ROI arguments I read quietly assume the first story, usually without noticing that they’ve assumed anything. The paper’s own negative results are worth the same attention, and they’re in the body rather than the abstract: multi-turn routing shows no consistent gain over single-turn, because the extra decomposition rounds cost real money for unreliable benefit.

Then there’s the part that needs a second paper to see. In the same week, another group rewrote 4,200 full manuscripts derived from 120 anonymized ICLR 2026 submissions, varying six rhetorical dimensions while holding the reported scientific content fixed, and measured what LLM reviewers did. The scores moved. Two independent teams, one week, one message: the LLM judge is measuring something other than what you think it’s measuring. Stack that against the routing result and the implication gets sharp. Tuning a router against an LLM judge is a two-stage proxy — the judge approximates human preference, the router optimizes the judge — and those two errors compound rather than cancel. A router will find the judge’s rhetorical soft spots exactly as reliably as it finds the big model’s capability holes. It is the same search, pointed at a worse target. LLMRouter shows the seam directly: under a simulated persona judge the personalized track ranks GMTRouter 68.78% / PersonalizedRouter 67.86% / best user-agnostic baseline 66.40%, while a deployment in front of real users put PersonalizedRouter at 83.05% preference-matching. Different population, different ordering, not a like-for-like number — and that last clause is the whole point.

The cheap countermeasure isn’t a better judge, because a better judge is the expensive countermeasure and it still leaves you one proxy deep. It’s a different audit axis, borrowed from an argument about hybrid interpretable models: audit coverage, not accuracy. Not how well the aggregate scored, but which queries got deferred. A router with clean headline numbers and a pathological deferral set is precisely the failure a leaderboard is structurally unable to see, and it’s the same shape as the β problem — a surface reported as a point, with the reader left to assume the missing dimensions are flat. One caveat I’m keeping rather than filing away: the 14.6% is measured against a fixed-model baseline, which is a soft target. The comparison that would actually settle whether learned routing earns its complexity is learned router versus a well-tuned heuristic cascade, and as far as I can tell that comparison isn’t in the paper. It rarely is. Beating the thing you were obviously going to beat is how a field spends a year measuring its own enthusiasm.