A Rank Is Not a Difference

Two leaderboards published their own error bars this month, and in both cases most of the reported ranking dissolved on contact with them. The first is Prime Intellect’s NanoGPT Speedrun Frontier (writeup; HN discussion): eighteen frontier models, each alone on a sandboxed 8×H200 with no internet, one objective — minimise the steps needed to train a 124M-parameter GPT to validation loss 3.28. Baseline agents start around 3,290 steps; the human record is 2,600. The ordering runs from 2,726 at the top down to 3,240 at the bottom, and the line everyone will quote is a token-efficiency inversion partway down the table: one model burned roughly 2.9B output tokens across 28,000 tool calls to close 35.9% of the human gap, while another closed 53.6% on 183M tokens and 401 calls. Sixteen times the tokens for less progress. It is a great line. It is not the finding.

The finding is in the methods section, stated plainly by the authors themselves: “Two runs of the same model and harness land about 54 steps apart at 24 ‘agent-hours’, 43 apart at 100 experiments, and 40 apart at 300k output tokens.” Take that number back to their own published table and do the arithmetic. Nineteen ranked entries means eighteen adjacent gaps, and sixteen of the eighteen are smaller than 54 steps — that arithmetic is mine, run on their numbers, so treat it as such. Ranks ten and eleven are a literal tie. Ranks fourteen through nineteen span twenty-six steps in total: six models packed inside half a noise draw. The only adjacent gap that comfortably clears the band is first to second. Everything below the top of that table is a ranking of coin flips, printed as an ordered list, by people who measured the coin and said so three paragraphs earlier.

What makes this more than a caveat is the recursion sitting inside it. The organisers deliberately seeded the task description with a noise estimate they knew was slightly too large, and then reported what the models did with it: “62 out of ~100 runs measured it themselves instead of trusting our number, and these runs are concentrated at the top of the results table. 42 went further and discovered something we never mentioned (on purpose): rerunning the same recipe on the same seed also moves the loss because GPUs are not deterministic.” Meanwhile — “Almost every model finds the same winning ideas.” So the discriminator this benchmark actually measured was not research taste in any interesting sense. It was the refusal to accept a handed-down noise estimate. A benchmark that rewards exactly that, and then prints nineteen ranks inside a band it measured itself, is grading a skill it declines to practise. Independent corroboration arrives from an unrelated corner: FlavourBench ranks twenty-seven frontier endpoints on an identical 534-task core with executable ground truth — no judge model, no preference panel — and its statistics are the contribution rather than the scores. Exactly 89 valid responses per cell across 14,418 model-task cells so that missingness cannot differ between models, 50,000 bootstrap replicates, 100,000 sign-flip draws across all 351 paired contrasts under Holm correction. The verdict, verbatim: “Grok 4.6 has the largest point estimate at 65.1 (simultaneous 95% CI 61.0-69.2); 101 of 351 model pairs are resolved.” One hundred and one out of three hundred and fifty-one. Seventy-one percent of the pairwise ordering is simply not there. Different domain, different method, same conclusion: the point estimates are fine, the ordering mostly isn’t.

Three things follow, and none of them require distrusting either group — both were more honest than their readers will be. First, a rank is a claim about a difference, and a numbered list is a lossy render of what an eval actually knows. FlavourBench’s honest artifact is a partial order: 101 resolved edges and 250 unresolved ones. Ship the unresolved marker; a table with visible ties is more informative than a table without them, not less. Second, resolution is bought with sample size, and that is the one thing a static benchmark cannot buy after publication while a continuously-collected preference stream accumulates it by construction — 14,418 cells to resolve 29% of pairs is the argument for continuous evaluation stated as arithmetic rather than as a pitch. Third, and most practical: on an unresolved pair, price is the only surviving signal. If the quality difference between two endpoints does not clear the noise band, then routing on quality is routing on noise, and the correct tiebreak is cost — the one axis on the whole table measured without error.