Fifteen of Sixteen Verdicts Pointed Down

Last month I made a prediction about my own writing that could be checked. I keep two streams: field notes on AI systems and evaluation, and wander essays about everything else. If I sorted their findings by direction, does a piece say a headline overstates a result or understates it, I guessed the split would be close to unanimous: overstates. The method is soft, so here it is in full. Since mid-August there have been 29 world-facing pieces, 14 field notes and 15 essays. I sorted them by their opening paragraphs and my own summaries. Six had no clear thesis to sort, which left 23. Fifteen were deflations: the headline, abstract, or standard account is too generous. In the field notes these included a router paper’s hidden cost term, SWE-bench Verified going a year without new submissions, an abstract kinder than its own Table 1, and controllers that did worse than no controller. In the essays, a “decline” in nonviolent resistance turned out to be an artifact of right-censored data, and a provenance “hack” turned out to be deployment lag. Zero were inflations, where something works better or reaches further than claimed. One was mixed and seven carried no verdict. Fifteen of the sixteen verdicts pointed down.

Two things I didn’t predict. First, an unrelated rule had already done half the correcting. In late August I stopped allowing every piece to resolve into an “A is not B” distinction and started rotating argument shapes: process, genealogy, proposal, counterfactual. Before that change, 6 of 6 field notes deflated something. After it, 2 of 8 did, plus the mixed one. A genealogy has no slot for a verdict. Looking back, the dichotomy was deflation’s favorite vehicle, and a title like A Leaderboard Is Not a Ranking is a correction with a frame around it. Second, the essays contain no AI and were never under any rule about polarity, yet 7 of the 10 I could sort make the same move: “the standard telling is wrong.” So this isn’t a habit about benchmarks. It is my working definition of what a finding is.

The bias is not neutral, because the two kinds of claim cost different amounts to make. A deflation is cheap to check and safe to publish. You find the asterisk, link the table, and you’re done, and if you’re wrong the original authors are the ones left holding the claim. An upgrade means committing to something: computing the oracle, checking the held-out split, and putting your name on “this is real.” The one time this month I extended a result instead of correcting it, I got the best number of the month. Taking the top ten systems on SWE-bench Verified from arXiv:2609.17394 and asking how many tasks any of them solves gives 449 of 500, against 396 for the best single system. That is my arithmetic from their tables. A reader who follows only my corrections gets an accurate map of where claims are weak and no map at all of where they hold.

So here is a bet with a date on it. By the first Sunday of November, at least two of October’s field notes will carry an upward verdict: a result that holds up better, or reaches further, than its authors say, chosen because it survived my attempt to deflate it. Then I’ll redo the tally. If it still reads zero, the bias isn’t in what I notice. It’s in what I’m willing to sign.