Who Is Measuring Pass@16?

I passed on this paper twice, both times on the strength of its abstract. Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO (arXiv 2608.27351 — Yunpeng Ba, Zhi Zheng et al., submitted 27 August; SUSTech / NUS / Huawei Noah’s Ark / CityU HK) names no model, no benchmark and no number in its abstract. Sixteen upvotes on HF daily papers, and twice I wrote “no result” and moved on. The third time I pulled the PDF instead. Thirty pages. The number that matters is not in the abstract, and it is not the number the title is selling. It is this: in 15 of 18 comparisons, GRPO post-training left the model below its own base checkpoint on both Pass@16 and Pass@32.

The setup is ordinary. Two epochs of GSM8K on Qwen2.5-1.5B-Instruct, Llama-3.2-3B-Instruct and Qwen2.5-7B-Instruct; six evaluation tasks each (GSM8K plus five held-out — CSQA, HotpotQA, Countdown, GPQA, MBPP); three models × six tasks = 18 comparisons. GRPO wins Pass@1 essentially everywhere. On Llama-3.2-3B the six-task average goes 44.1 → 47.1. Over the same run, average Pass@32 goes 78.6 → 77.0. On GPQA specifically, GRPO buys +3.9 points of Pass@1 and pays 2.0 points of Pass@32; Evolution Strategies on the identical budget lands +2.1 and +5.1. The authors track token-level entropy through training and watch it fall under GRPO while Pass@16 and Pass@32 sink beneath the base model — entropy collapse with a receipt attached.

Now the dead ends, which matter more than the headline. ES’s own gains are thin. At 7B the effect nearly vanishes: ES posts 59.6 average Pass@1 against the base model’s 60.1 — it loses — and its Pass@32 edge is 83.1 vs 83.0, which is nothing. And in the paper’s Hard Setting (DeepSeek-R1-Distill-Qwen-1.5B, one epoch of DeepScaleR, AIME24/25 + AMC23 + MATH-500) the whole story softens: GRPO takes Pass@1 47.7 → 52.9 and also nudges Pass@32 up 77.4 → 78.0. So “GRPO destroys coverage” is an easy-task, cheap-reward finding, not a law. The best configuration in the paper is neither optimizer alone but a sequential ES→GRPO run: 52.3 Pass@1, keeping most of GRPO’s gain, at 79.2 Pass@32, the best number in the table. Two further honest negatives are worth having: two-point ZO estimation, which borrowed intuition from SST-2-era zeroth-order work says should help, provides no advantage on regenerated reasoning rollouts; and the authors directly contradict Abdi et al. (2026) on ES causing catastrophic forgetting, arguing that result rested on one model, one task and one held-out benchmark, and is indistinguishable from train-set overfitting.

The mechanical detail I did not expect: ES moves the weights 40.7× to 44.1× farther from base in relative whole-model L2 than GRPO does — and it doesn’t matter. Zero out every update below 1.5×10⁻³ and you have deleted 77.6–93.0% of all nonzero changes while Pass@1 stays flat. The survivors concentrate in LayerNorm and attention projections; 72 of the top 100 updates in Llama-3.2-3B are normalization parameters. GRPO’s largest updates are 16–48× smaller and sit entirely in token embeddings and the LM head. Two optimizers, one objective, structurally different parts of the network edited.

Which leaves the part nobody prices. Pass@K is the operating metric for anything that retries, samples n, or falls back across a pool of models — such a system is buying coverage, not single-shot accuracy. If the default post-training recipe systematically trades Pass@K for Pass@1 on easy tasks with cheap rewards, then a large share of the RL-tuned checkpoints in circulation have been quietly optimized away from the property that multi-sample systems depend on, and the loss is invisible on every leaderboard, because leaderboards score one sample. The ask is cheap: when you evaluate a checkpoint, log Pass@16 next to Pass@1 and diff it against the published base model. That delta is a real, measurable property, and at the moment almost nobody reports it. Code: github.com/yunpengba7/understanding-es.