SWE-bench Verified, Read Per Instance

On 15 September, Fengshuo Liu and four co-authors posted “Coding Agents Have Converged”. It audits 254 SWE-bench submissions without running a single model — it only re-reads the per-instance verdicts the leaderboard already publishes. Their case, at full strength: on Verified, the two leading entries each resolve 396 of 500. Across the top ten, 285 instances are solved by all ten and 51 by none, leaving 164 that can tell any of them apart; for the top two, just 36. The rest count toward every score and inform no comparison. Successes are nested (median nesting 0.935 against a score-implied 0.774), so these are not specialists solving different problems. An exact paired McNemar test on each adjacent pair in the top thirty separates none of the 29 at α=0.05; the median gap is one instance. The control is what makes this more than a complaint: on the 2,294-instance Test split, where the top 24 span 0.2% to 52.6%, 14 of 23 adjacent pairs do separate. And the ranked number belongs to a model–scaffold pair, not a model: within-model scaffold ranges reach 29.8 points, more than three times the 8.8-point spread of the whole top thirty.

The most useful part is a pricing rule. A paired test needs roughly k ≥ 1.96²(b+c)/(b−c)², where b and c count instances only one system solves. For the leading adjacent pairs the median multiplier is 52× — about 26,000 more instances “of the same character.” But for a pair with 54 discordant instances, new items that lean one way at a 0.2 imbalance need only about 900. Hence their curation rule: admit a task by what it adds to the discordant budget. The obvious fix, retiring the 285 easy instances, doubles the visible spread (8.8 → 18.7 points) and reorders three pairs while separable pairs stay at 0 of 29 — a wider scale with zero added power.

Here is where the same data reaches further than the paper takes it. The authors read convergence as bad news for ranking, but the matrix also prices a router. An oracle that picks, per instance, whichever top-ten system succeeds resolves 500 − 51 = 449, or 89.8%, against the best single entry’s 79.2% — 53 instances of headroom. Over just the top two, it gets 396 + 18 = 414 (82.8%, +3.6 points). That gain lives entirely inside the 164 discordant instances, and there is a catch the paper itself flags: one run per submission, so run-to-run variance is folded into those 164. A router trained on them may be learning coin flips. Before anyone builds a coding-agent router from leaderboard traces, the first experiment is re-running the discordant set several times and measuring how much of the 53 survives. The authors released their pipeline; that re-run is cheap.

Preference arenas face the same arithmetic on worse terms. They are paired too — two models, one prompt — but each prompt usually meets only one pair, so they lack the all-systems-on-all-items matrix that made this audit possible. Ties and “both fine” votes are an arena’s degenerate instances, and the acceptance rule translates directly: tilt prompt sampling toward prompts where frontier models disagree, and publish the effective count of decisive votes beside every rating gap. The paper’s advice for internal model selection — evaluate the (model, your harness) pair, never a transferred ranking — bites harder there, where the “harness” is a chat template and system prompt that rarely get logged; a same-week paper finds the chat template alone shifts how eight 3–9B open models talk about themselves. If repeated runs show the 164 are stable and a learned router captures most of the 53, convergence becomes a routing opportunity as much as a ranking problem.