From HumanEval to SWE-Duel

On October 8, four researchers at Singapore Management University (Wei Minn, Zhensu Sun, Lwin Khin Shar, David Lo) posted SWE-Duel, a coding arena in which the contestants write the exam. A “Red” agent implements a feature with tests, then plants a subtle bug along with tests that trigger it. A “Blue” agent reviews the PR and has to fix it, and then the two swap roles. The run covered 11 model–harness pairs, 15 repositories, 5 languages and 1,500 turns. GPT-5.5 with Codex CLI repaired 98.3% of planted defects, and Claude Opus 4.8 with mini-swe-agent was the strongest attacker, winning 27.3% of its Red turns. The result behind the “self-scaling” in the title is that tasks written by newer models beat a fixed reference solver 42.9% of the time, against 4.0% for tasks written by older models. Stronger contestants write harder exams.

The paper makes more sense once you trace who has held the oracle in coding evaluation over the past five years, because each step reacted to the one before. In 2021 lab staff wrote it: HumanEval was 164 Python problems handwritten at OpenAI, so the authors were also the graders, and it saturated within about three model generations. In 2023 repository maintainers wrote it: SWE-bench took 2,294 real issues from 12 Python repos and used each merged PR’s own tests as the oracle. That answered the toy-problem complaint but inherited the maintainers’ habits, like tests that check details the issue never mentions. In 2024 hired auditors wrote it, when 93 professional developers screened 1,699 samples down to the 500 of SWE-bench Verified. The same year, LiveCodeBench let the calendar do the guarding by scoring each model only on problems published after its training cutoff, and Copilot Arena let users write it: a developer accepting one of two completions in the IDE counts as a vote. Then, this year, the audited set wore out. OpenAI stopped reporting SWE-bench Verified after finding material flaws in 59.4% of 138 of its hardest problems, along with signs that frontier models could reproduce reference fixes from memory. A one-time human audit doesn’t stay clean.

SWE-Duel is the next move in that sequence. Its tasks don’t exist until the duel starts, so they can’t leak, nobody has to be paid to write them, and their difficulty rises with the players. Each generation handed the oracle to whoever the previous one had stopped trusting, and this one hands it to the contestants. The trust doesn’t vanish, though; it collects in one place, the validity filter. A Red PR counts only if it passes the feature tests and the existing tests, fails only the bug tests, and can be repaired by the same Red agent in a fresh session. That last condition stops an attacker from planting an unfixable bug, but it’s a self-check: the model that wrote the bug decides whether the bug is fair. Blind spots shared within one model family can slip through, and an attacker that’s good at bugs only it can find and fix is exactly what a self-repair test lets in. Sakana’s MASS notes the same homogeneous-loop risk in its limitations.

That suggests where the next step in the lineage might go. In a self-scaling arena the expensive human work moves out of writing tasks and into auditing the filter, so sample both accepted and rejected PRs. Requiring that a model from a different family can repair each planted bug would close the main loophole cheaply. Rating attack and defense separately matters too, since a 98.3% repair rate and a 27.3% attack win rate measure different skills that a single Bradley–Terry score would blur. And the rejected PRs are worth logging rather than discarding: Terence Tao’s Math 2.0 slides ask AI evaluations to report negative results, and an arena’s invalid tasks are exactly that record.