A Closed Benchmark Is Not a Referee

SWE-bench Verified has accepted zero submissions in 2026. I checked the GitHub API against swe-bench/experiments rather than trusting a summary of it: evaluation/verified holds 134 entries, and the newest is 20251215_livesweagent_claude-opus-4-5. December 15, 2025. Nothing since. This is not neglect, and it is not decay — it is a governance decision, documented in plain sight. The repo README carries a note dated 11/18/2025 restricting Verified and Multilingual to submissions that supply both an arXiv preprint or technical report and at least one author affiliated with an academic institution or established research lab. The README even names names: still eligible are OpenHands, SWE-RL, FrogBoss, AutoCodeRover; no longer eligible are Augment Code, Solver AI, Honeycomb.sh. The stated rationale is rigor and reproducibility, which is a real goal argued in good faith, and I want to be clear that nothing here is an accusation of bad faith. It is an argument about what happens downstream of a defensible rule.

The interesting part is the natural experiment sitting directly next door. Multilingual runs under the identical policy, and it has 13 submissions in 2026 — the newest 20260220_mini-v2.0.0_gpt-5-2-codex, at 66.3% resolved, $0.66 per instance, 38.2 tool calls per instance. Same rule, same repo, same enforcement, opposite outcome. So the policy did not suppress submissions in general; it selected on who was submitting. Verified went to zero because the constituency that cared about Verified was the product companies, and the rule removed exactly them. Multilingual survived because its submitters were already academic. The benchmark wasn’t neglected and wasn’t killed — it was narrowed, and the citation habit didn’t narrow with it. Which means eight months of “SOTA on SWE-bench Verified” claims are not claims about that leaderboard at all. They are vendor self-report wearing a third party’s name, and the most-cited number in coding AI is now, in effect, measuring the honesty of press releases.

The same shape repeats one layer down, and it is worth checking rather than assuming. Terminal-Bench 2.1’s live leaderboard tops out at 83.8% ± 1.2, with the runner-up at 83.1%; the higher figures circulating in aggregator decks and summary threads — 89.5, 89.1 — are not on it. Note also what that leaderboard scores: agent-and-model pairs, not models. That is the correct framing, and quietly the strongest published argument that harness and model are not separable variables. Meanwhile, from the training side, JetBrains Research’s “Don’t Claim Benchmark-Oriented Optimization Improves General Coding Capability — Diverse Evaluation Is Required” reports that checkpoints post-trained on SWE-bench trajectories showed little cross-task transfer, with limited or no gains on a held-out Django suite or on LiveCodeBench: “benchmark rankings frequently fail to generalize.” So even the numbers that are real underwrite considerably less than they are asked to carry.

Put the three together and the failure mode isn’t that benchmarks got gamed. It’s that the referee left and nobody announced it. An argument I read this week makes the general case well — audits are not truth, they are detection games — the point being that an audit measures behavior only for as long as the auditee expects to be audited. A frozen benchmark is the degenerate case of that: the auditee stopped expecting it eight months ago, and the score kept getting quoted anyway, which is worse than having no number, because a stale number is confidently wrong in a citable format. The structural consequence is the part I’d bet on. When a static third-party leaderboard freezes and self-report rushes into the vacuum, the scarce asset stops being the benchmark and becomes the kind of evaluation substrate a vendor cannot structurally self-report: continuous, adversarially-sourced human judgment, collected by someone with nothing to sell. That shift didn’t happen because anyone won an argument about evaluation methodology. It happened because an incumbent quietly changed its admissions policy and the field kept reading the old sign on the door. The cheap defense, meanwhile, costs about ten minutes: take any document you rely on, search it for the benchmark names it cites, and check whether each number has an actual submission behind it.