Line One, Every Time

GVS5H (HN discussion) makes a strong claim. Fresh instances of one cheap model, coordinating only through files on disk (plan.md, notes.md, solution.py), beat Claude Fable 5 on the 100 newest hard LiveCodeBench problems. Qwen3.8-Flash-Next with a manager loop scores 93.0, and Fable 5 in a single call scores 90.4. A pass costs $5.76 against $61.11. I wanted more than the README, so I followed the numbers into the paper source. The first stop was cost, and it holds up. The cost table bills every retried and discarded attempt at list price. Per solved problem, it comes to $0.062 against $0.68.

The second stop was why the gains vary so much, from +25.6 points for Qwen3.8-27B down to +3.6 for DeepSeek-V4.1-Flash. The paper’s explanation is truncation. 30% of Qwen3.8-27B’s single calls hit the 128k output cap. In eight of them the model ran away and used up a 250k budget without writing any code. On one problem it repeated a single phrase 7,743 times. The ledger bounds each call and writes progress to disk before the reasoning degrades. That mechanism is real, and it shows what the method actually buys: it rescues models that can’t stop themselves. Kimi-K3 with reasoning off goes from 33 to 76. With reasoning on, it starts at 86 and stays at 86. Qwen3.6-35B gains 18 points with reasoning on and loses 8 with it off, because its planning stage argued it out of a correct optimisation.

The third stop was the grader, and it changes the story. In an appendix, the authors report something they found while reading transcripts. LiveCodeBench’s evaluator runs submissions in-process with a mocked stdin, and the mock’s binary readline() is inputs.split(b"\n")[0]. It returns line one on every call. That line is still on main. Any program that reads input with sys.stdin.buffer.readline() fails no matter how correct it is. The more common sys.stdin.buffer.read().split() works, which is how the bug went unnoticed. The damage follows coding style, not skill. Of 5,425 outputs containing code, 371 use the broken idiom: 86% of those were scored wrong, against 18% of the rest. Fable 5 never writes it, so its correction is zero. GPT-5.6-Terra writes it often, so its single call rises from 69.8 to 80.8 once regraded (regrade table). The fix also lifts the cheap-model arms, and that changes the headline. On the stock grader, Qwen3.8-27B with the manager scores 84.4 to Fable’s 87.4, and Flash-Next scores exactly 87.4. The two “beats Fable 5” results are a loss and a tie. To make it worse, the scaffold’s own verifier ran candidates as real subprocesses, where readline() works. So the loop’s only outside signal kept approving answers the grader then rejected. The fix, issue #160 and PR #161, has been open since September 30.

Some questions stay open. There is no Fable 5 manager arm, so we don’t know whether a frontier model gains from the scaffold too. As the top HN comment notes, LCB-hard is contest code, not repository work. Still, the investigation leaves three habits that carry over to any benchmark. First, a published score can carry an error that depends on a model’s style. Here that error was 8 to 12 points for one model family and zero for another, larger than most leaderboard gaps. Second, there is a cheap way to catch this. Bucket outputs by a surface feature (an idiom, a format, a length, a language) and compare failure rates within each model. If a feature predicts failure across models at 86% against 18%, the grader is the problem, not the models. Third, when an in-loop verifier and the final grader disagree, record it. Here the disagreement was the bug report, and it sat unread in the transcripts until someone opened them.