A List Price Is Not a Bill

Two things landed this week that look unrelated. On 19 August Replit shipped Free Mode: unlimited lightweight AI work on GPT-5.6 Luna for Core and Pro subscribers, with automatic escalation to GPT-5.6 Sol when a request needs heavy reasoning — complex codegen, multi-file debugging — and a warning to the user before it switches (OpenAI; Fortune). It is the cleanest public production instance of model routing as a business: a cheap tier absorbs the long tail, a policy promotes the hard turns. It exists because OpenAI cut Luna’s price roughly 80% on 30 July, and Luna already ran about 80% below Sol. The entire offering is an arbitrage on an intra-family list-price delta of about 5×. Meanwhile, Codex issue #37674 — opened 9 August, still open — reads “Native Bedrock Codex GPT-5.6 Sol lacks explicit cache controls, producing high cache-write spend.” Four days, 3,656 requests, 171.94M cache-write tokens: $1,182 of a $1,386 bill, 85% of spend, on cache writes. One local session logged 6.709M cache-write tokens and zero cache reads. The bisect is clean — write-to-read ratio went 0.08 → 8.84 upgrading v0.146.0 to 0.147.0, and back to 0.025 on rollback. OpenAI’s reply on 20 August: a newly enabled web-search tool caused the miss; disable the tool. That same day Claude Code v2.1.237 shipped a two-line changelog whose first line is “Fixed prompt caching for sessions using an LLM gateway or custom base URL” (changelog). The connection is that the first story is priced on a number the second story proves isn’t the bill.

DeepSeek makes the gap legible, because it publishes cache-hit and cache-miss input prices separately (pricing docs). Off-peak, per million tokens: deepseek-v4-flash costs $0.007 on a cache hit and $0.22 on a miss; deepseek-v4-pro costs $0.022 on a hit and $0.66 on a miss. That is a ~31× spread between hit and miss on the same model at the same tier — six times larger than the ~5× cross-model spread the routing business is monetizing. Follow it one step further and it inverts: the expensive model on a cache hit ($0.022) is ten times cheaper than the cheap model on a cache miss ($0.22). Routing down can cost an order of magnitude more, if the downshift breaks the prefix.

And it does break the prefix, structurally. Armin Ronacher’s What Is Reasoning supplies the generalization almost in passing: reasoning traces “really are just text,” and “reasoning effort is baked into the system prompt… This also explains why changing the effort invalidates the KV cache.” Every lever a router pulls is a prefix mutation. Switch model, new prefix. Switch effort tier, new prefix. Enable a tool for this query but not that one, new prefix — which is literally what bit Codex. Route through Bedrock or Vertex instead of first-party and you get different cache semantics your harness may not even plumb through. So the act of routing carries a cost the routing calculus doesn’t price, and it is billed at the write rate, the expensive one. The error is not noise. It is systematic and one-directional: it always makes routing look cheaper than it is, because switching is the thing routers do. The harder you chase the spread, the more prefix rewrites you buy.

Three consequences worth acting on. First, a cost coefficient computed from list prices is the wrong β: real cost-per-query is a function of your harness’s prefix stability, not of the model you picked, and two routers with identical policies can produce very different bills. Second, switching needs a hysteresis term — an escalation is only a win if the quality gain exceeds the price delta plus a full prefix rewrite at cache-write rates, so cheap-to-expensive one-way per session beats flapping. Third, you cannot claim to manage inference cost without per-turn cache read/write telemetry. Nobody had it; that is why a 4× bill ran for days at two vendors simultaneously, and why the Codex reporter’s own feature request asks for exactly that. Replit got one thing instinctively right that nobody framed this way: they made the model switch a visible event. They did it for billing transparency. The better reason is that a switch is a cache-invalidation boundary — and boundaries you can’t see are boundaries you can’t price.