A Regret Ledger for Decision Models

In about a week the “decision model”, a model that returns a typed choice with a probability instead of prose, went from a single product to a crowded market. On October 1 Cloudflare released Clef and Clef-flash under Apache-2.0, built on Qwen 27B and 9B bases. On BANKING77 Clef reports 94.20 macro-F1 against Jev’s 79.74, and Clef-flash reports a median latency of 38.8 ms against Jev’s 524.1 ms. Pienomial’s At0M has 60M parameters, answers in 16 ms on a T4, and reports 78.9% on a 2,000-decision typed benchmark against Jev’s 74.0%. Pi 1.0 ships “virtual models” in which a decision model picks a frontier model for each turn. The router is now a part you can buy. The scoring hasn’t kept up, though. Every number above is accuracy or F1 on a public suite, and the headline index is maintained by the company that makes Jev. A buyer needs to know something else: how much would routing this model’s way cost on my own traffic? Here is a proposal for the instrument that should answer that, a regret ledger.

First, replay real traffic instead of running suites, and score by regret. Take a few thousand logged requests whose outcomes are known: which arm served the request, whether it succeeded, and what it cost. Pairwise preference logs fit this format naturally. Give every candidate the same requests and the same menu of arms, and score each by expected regret under a cost matrix the buyer writes: the dollars wasted sending easy prompts to the most expensive model, and the quality lost sending hard ones to a 9B model. A Brier score counts both errors the same, but routing costs are not symmetric. Second, report abstention in its own column. The Cloudflare launch table has a row the post doesn’t discuss. On CLINC150+OOS, an intent benchmark with an out-of-scope class, Clef scores 97.43, Jev 89.27, and Clef-flash 66.77. The fast, cheap model, which is the one you would actually put in front of every request, loses 30 points when the right answer is “none of these.” In a router, “none of these” is the fallback path. It should never be averaged away.

Third, count flips, not net gain. SAKIKO (Li & Li, September 28) audited interventions on tool-use decisions across seven LLMs and found one with a +55 net gain that corrupted more than half of the baseline-correct decisions it touched. Swapping one router for another has the same structure. A +2-point net made of 400 fixes and 380 breaks is a very different migration from one made of 20 fixes and no breaks, so the ledger reports both counts against the incumbent. Fourth, report a learning curve instead of a single point. The sharpest objection on Hacker News came from a commenter who argued that “if you can neither pass in a prior nor fine tune for your system, you will get an output that may be almost meaningless.” So each model should be scored at 0, 100, and 1,000 labeled in-domain examples. A model that wins at zero and stays flat should lose to one that starts lower and learns.

An evaluation proposal should say how it could fail, so here are two kill conditions. First, the ledger has to disagree with the public index somewhere. If it ranks every pair in the same order as the vendor’s index, it adds nothing and should be dropped. Second, offline regret has to predict online regret. Run the top two candidates in shadow mode for two weeks and check that the gap the ledger predicted has the right sign and roughly the right size. If it fails either test, it is just another leaderboard. If it passes both, routers would finally be judged on what they cost the people who deploy them.