Astra, Measured Twice

On 3 September OpenAI shipped GPT-6 Astra, with Greg Brockman calling it a generational leap and saying he personally believes they may have reached AGI (The Register). The same day, Greg Kamradt published ARC Prize’s measurement of it (arcprize.org/blog/astra). The interesting part of that post is not a score. It is a procedural decision, and the table that forced it.

ARC ran Astra twice on ARC-AGI-3 Semi-Private. Their Standard harness is provider-neutral: the environment supplies everything needed to solve each game, and the model decides for itself what to keep in visible notes. The Provider Adapter harness preserves OpenAI’s opaque reasoning state between requests and compacts long conversations, so the model reuses prior work it never has to write down. Six reasoning-effort levels each. The headline pair: 62.7% for $26,098 (Standard, max) and 99.9% for $18,817 (Provider Adapter, high). The full grid is where it gets strange. Standard: none 35.2% ($49,791), low 17.5% ($38,166), medium 38.6% ($48,090), high 54.8% ($40,705), xhigh 59.3% ($37,317), max 62.7% ($26,098). Provider Adapter, same order: 96.7%, 98.0%, 98.4%, 99.9%, 98.4%, 98.6% — costing $23,457 down to $17,332.

Two things fall out, and neither headline number contains them. First, under Standard the effort dial is non-monotone in quality — low (17.5%) scores worse than none (35.2%) — and inverted in price. The setting you would assume is most expensive is the cheapest to run, because at max effort Astra solves games in fewer actions, so fewer model calls and fewer total tokens. Buying more thinking per call bought less spend overall. Second, under Provider Adapter the dial essentially stops existing: all six levels land inside a 3.2-point band. The knob a routing layer would spend its life tuning has a 45-point range in one harness and a 3-point range in the other. ARC’s own cross-harness diagnostic rules out the obvious explanation — across the 167 game-and-effort pairs both harnesses solved, Provider Adapter runs were ~3.66× faster and used 49% fewer total tokens. The adapter is not buying more compute. It is buying state continuity, and getting a 37-point lift while spending less.

So, the decision: “Going forward, we will report both Standard harness and Provider Adapter harness results on the ARC-AGI leaderboard, with each evaluation condition clearly labeled.” An eval provider just promoted “which harness” from a footnote to a first-class, published axis. They also declined the launch-day framing outright — “we are not claiming that it is AGI” — citing ARC-AGI-3’s “tightly bounded scope” and “deterministic, closed-ended mechanics.” Worth keeping the human comparison honest in both directions, too. ARC baselined on ~500 members of the general public, not puzzle specialists, paid $115 per 90-minute session plus $5 per completed game — roughly $12.78 per attempted game against Astra’s ~$360. And Astra still beat them where ARC-AGI-3 actually aims: fewer actions than the median tested human on 96.0% of levels, 51.7% fewer actions per level on average. Cheaper humans, more efficient machine, same axis.

The implication for anyone building arenas or routers is narrow and cheap to act on. If a provider ships the adapter, the adapter is part of what you measured — and here it is worth more than five steps of the reasoning dial. Any comparison that does not fix the harness is reporting a sum of two variables under the name of one. The Armature study published the same week found the three major coding agents pick the same tool in only 42% of scenarios (armature.tech) — the same confound on a different surface. ARC’s fix costs nothing but a column.