Leaderboard Noise Lives in the Prompts

Krishna Ayyagari’s Pinned and Still Unstable (v2 PDF) is a cheap study, roughly $5,000 of cloud API calls, and its tables are more useful than its abstract. The setup: 4 LLM judges (gemini-2.5-pro, claude-haiku-4-5, claude-sonnet-4-5, claude-sonnet-4-6), 10 candidate models, 3 benchmarks (500 Arena-Hard prompts, 805 AlpacaEval 2 prompts, 80 MT-Bench prompts), and every verdict re-run 10 times with the judge snapshot pinned and temperature at 0. Even pinned, the judges wobble. Mean per-item flip rates on Arena-Hard run from 0.13% for Haiku to about 5% for Gemini and Sonnet-4.5, and on MT-Bench Sonnet-4.6 reaches 14.71%. That is a 40× spread, and it sits entirely in what the paper calls the waver fraction: how often a judge ever changes its mind. About 12.5% of Gemini’s Arena-Hard cells flip at least once. Once a cell does waver, though, the conditional flip rate is 34–40% for every judge, which is roughly what you would expect if close calls are spread evenly from 1-in-10 to 5-in-5 splits. The hard items look alike to all four judges. The cheapest judge in the set is also, by a wide margin, the steadiest.

The more surprising number is in the noise decomposition. Of the 9 adjacent rank pairs on Arena-Hard, 5–7 can’t be statistically separated from prompt sampling alone, depending on the judge. Judge re-run noise by itself leaves 3–4 unresolved. Combine the two, and across all 7 judge × benchmark cells the count goes up by zero in six cells and by one in the seventh. In other words, a perfectly deterministic judge would still leave most Arena-Hard neighbours indistinguishable. The floor under “model X moved up one place” is set by how many prompts were asked, not by how stable the grader is.

Head-to-head claims tell a different story. Of 15 expected pairwise orderings, 12 survive all 10 re-runs of Gemini, but only 8 survive every judge. The 7 failures split three ways. Three are cross-judge disagreements: only Gemini ranks Gemini-2.5-Pro above Qwen3-235B, which looks like self-preference. Two are re-run flips: Llama-4-Maverick against Llama-3.3-70B flips on 4 of 10 runs, at raw win rates of 0.396 versus 0.393. Two are unanimous contradictions, and one of those (Opus 4.5 versus Qwen3-235B) is confounded by length, since Opus’s answers are 43% shorter.

Put a price on each fix and the paper reads as a budget sheet. Re-running the judge, which the paper recommends at R≥5, is the most expensive option and moves the leaderboard least. What re-runs actually buy is item-level information: they reveal which ~12% of verdicts are coin flips, and anyone building preference or routing data from LLM labels should keep those as soft labels rather than throw that signal away. A second judge family is the best value for pairwise claims; one extra family overturned 4 of the 12 orderings Gemini reproduced perfectly. And more prompts are the only lever that shrinks the adjacent-rank floor. The good news is solid too: for every judge, the top-1 model and each candidate’s win/loss sign against the baseline never changed across re-runs, so a leaderboard’s coarse shape is real. The caveats are real as well. The judges are 2025-vintage, the study used one cloud platform and has one author, and the benchmarks are fixed-baseline win rates rather than human Elo. Still, the practical rule is simple enough to apply to any board you read this week: report the judge, report a second family, and publish intervals for neighbours instead of ranks.