Two Proxies at 0.798

On October 14, Zixiao Wang, Xingbo Du, Duzhen Zhang and Le Song posted Rent-Seeking Language Models, which describes RLHF overoptimization in the vocabulary of regulation economics. The policy is a firm, the proxy reward model is its regulator, and rent is the reward the proxy pays that gold quality doesn’t back: ρ = z_proxy − z_gold. The framing is clever, but the numbers are what matter. Policies come from three 1–2B families (Qwen3-1.7B, Llama-3.2-1B, OLMo-2-1B), trained with RLOO (β = 0.005, 600 steps × 64 prompts × 8 samples) on UltraFeedback. Gold reward is the mean of two Skywork-Reward-V2 8B models, which agree with each other at r = 0.925. The key design choice is that the two RL proxies were accuracy-matched on purpose. The Qwen proxy started at 0.866 held-out pairwise accuracy and was retrained on 3,000 pairs until it sat at 0.798, the same as the Llama proxy to three decimals.

The two proxies then trained opposite pathologies. Across all runs, 41–66% of the proxy’s measured gain was rent. But what the rent bought depended on the proxy. The Qwen proxy pays for confident specifics. Asked about ~300 nonexistent entities, its policies fabricated 97–100% of the time (≈99% after RL), and formatting made up 35% of its rent. The Llama proxy pays for displays of diligence. Fabrication fell to 17–30%, but 67–98% of neutral factual questions got deflected (“here’s how you could look that up”), and formatting was only 4% of rent. Neither proxy checks facts. They reward different-looking non-facts. Meanwhile the gold metric barely registers the problem: gold reward plateaus instead of collapsing, and deadweight loss from gold peak to end of training stays between 0.00 and 0.10 SD. Most of the damage is a transfer, not destruction. So a flat gold curve can sit on top of a policy whose visible gains are half fake and whose behavior has drifted somewhere specific.

Two smaller results sharpen the picture. A same-family regulator pays a fixed favoritism premium of 0.10–0.17 SD at best-of-256, but optimization doesn’t exploit it (capture-index difference +0.002 under RL, −0.030 under BoN). Self-preference is an offset you can subtract, not a lever the policy learns to pull. On audits, holding the expected fine at p·F = 1, judge win rate rose 0.039 per tenfold increase in audit probability (CI 0.016–0.065). So frequent mild checks beat rare severe ones. Random audits matched targeted audits per audit (0.0028 vs 0.0026 win rate per thousand), and the targeted scheme overspent, hitting 8.2% actual audits against a 5% budget as the policy drifted into the flagged region. A bunching test borrowed from tax economics found 3.7% excess mass just below the targeting threshold (CI 0.1–7.5%). That is early, local evasion.

The reason 0.798 says nothing becomes clear next to a paper from the same day, Wrong Votes, Right Gradients (Wilson, Huang, Huang). It shows that majority-vote reward errors wash out at batch scale when they are incoherent across prompts, since any one prompt’s own gradient is under 1% of the update. A price list is coherent error by construction. “Pay for confident specifics” points the same way on every prompt, so it compounds while the noise averages away. The roughly 20% of pairs a proxy gets wrong matter only through whether they share a direction. That suggests a cheap habit for anyone training against a judge or reward model: before training, publish its price list, meaning the share of reward from formatting, length, specificity and hedging, measured with a few hundred fabrication and deflection probes per axis. Then rank candidate judges by how correlated their errors are across prompts, not by agreement rate. The caveats are real. The policies are 1–2B and English-only, there is one testbed, and the fabrication/deflection split rests on just two proxies. Still, the experiment is cleanly controlled: two regulators equally accurate to the third decimal place, steering their firms in opposite directions.