A Model Name Is Not a Configuration

The Embedder’s Dilemma (El Assadi, Muennighoff, Lee; COLM 2026) ran ten LLMs across six families against twenty-six embedding models, 118M to 14B, on thirty-seven retrieval tasks. The headline is the cost gap, and it is a good one: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) finish 0.4 points apart while the LLM costs up to 1,431× more — “USD 154 vs. USD 0.11 per benchmark pass.” That number will be quoted everywhere, and it is the least interesting thing in the paper. The buried line is in the ablation: “Reasoning tokens account for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improve retrieval quality for most models.” Read that as a routing result, because that is what it is. The single largest cost term in an LLM retrieval call is a knob inside the model — and on this task family the knob is non-monotone, in fact negative-value. You can take somewhere between 1.4× and 5× off the bill without choosing a different model at all, and quality does not drop; for most models it improves. That arithmetic is mine, from the paper’s stated range, so treat the interval as an estimate rather than a measurement.

This should be more obvious than it is, and the reason it isn’t is that routing — as nearly everyone builds it — is a choice over model IDs. That framing quietly assumes the model ID is a sufficient statistic for what you get and what you pay. Two things landed this week saying it isn’t. First, effort is neither observable nor stable. Claude Code users found that “high” was reading out as 10 — the value “low” used to display. Thariq, from the Claude Code team, confirmed the mechanism in-thread: “We sometimes test API serving configs in Claude Code before rolling them out, and one running now maps the numerical effort value differently… The scale isn’t 0-100, the number isn’t meaningful on its own, and the effort you selected is the effort you’re getting.” Take that entirely at face value — I do — and it is still the admission that matters: the mapping from requested effort to delivered compute is a server-side config the vendor mutates without a changelog entry. You cannot log what you asked for and call the configuration pinned.

Second, even with model and effort pinned, the serving stack moves the output. A careful Level1Techs writeup measured top-1 token disagreement of 0.717%–5.831% across Qwen 3.8 derivatives, NVFP4 hitting “~50% token flips by the time we reach 88k context,” and — the sharp one — INT4 KV cache producing reproducible tool-call failures where BF16 succeeded: “int8 kv-cache eventually managed to recover, int4 did not.” Degradation “appeared in clusters and varied with prompt content rather than increasing smoothly with context length.” Not linear, not on any spec sheet, and not one careful practitioner’s fluke in the direction that would be convenient — though it is one practitioner’s measurements, not a peer-reviewed result. Stack the three layers and the picture is unambiguous. Reasoning budget is a huge cost term. Reasoning budget is not stably observable. And underneath both, quantization scheme, KV-cache precision and attention backend are quietly rewriting the tokens.

Which is where this stops being an efficiency story and becomes a measurement one. For an arena or a preference dataset, the model name is the join key. If it is not a sufficient statistic for the served configuration, then every preference pair carries a latent nuisance variable — quantization, KV precision, attention backend, effort mapping — and that variable is correlated with provider, which is correlated with price. A router trained on that data learns serving artifacts and reports them as model quality. The cheap endpoint didn’t lose because the model was worse. Three things follow. The first routing decision isn’t which model, it’s how much reasoning — that is 28–81% of the bill, available before you touch model selection, and on retrieval-shaped work the answer is often “less.” Pin observed behaviour rather than requested settings: a canary prompt with a known output distribution catches a silent remap where a config log never will. And preference data should record the endpoint, not the model, or the serving stack sits inside the thing being sold as ground truth as an unmeasured confound. Artificial Analysis, for what it’s worth, now leads with Cost per Task, Time per Task and Output Tokens per Task alongside its nine-benchmark Intelligence Index. Right correction, one layer too high: it prices the task, not the configuration. And if procurement really is fragmenting from uniform GPU clusters toward heterogeneous silicon, then “the same model” means less every quarter, not more.