Where NAOD's 29 Percent Comes From

Anyone collecting preference labels at scale ends up mixing a few expensive human votes with many cheap LLM-judge votes, and the judges lean in systematic ways. A paper posted September 30, “Optimal Design for Active Preference Learning with Biased LLM Judges” (Du, Zhang, Zhu, Zhang), proposes NAOD, a method that decides which comparisons to buy, and from whom, while accounting for each judge’s lean. The headline is a 29.1% cut in regret on Chatbot Arena data. The number is less useful than the pipeline that produces it, so here it is one stage at a time. Stage one is the archive: a frozen set of 49,635 usable comparisons in 42,875 connected text clusters, split by cluster 15 separate times so that representation learning, label acquisition, and evaluation never see the same cluster. Stage two learns the lean. A judge’s preference is modeled as σ(Xᵀθ + Wᵀa). Here θ is the human reward you actually want, W is a fixed “judge-deviation” representation learned from history, and a is a nuisance term for how far, and which way, a particular judge drifts from people. There are 17 judges, and the text I read doesn’t name them.

Stage three decides what to buy. Budgets cover a 5×3 grid: human labels H ∈ {32, 64, 128, 256, 512} by judge records B ∈ {256, 512, 1024}. A Frank-Wolfe solver allocates comparisons to maximize policy-relevant information about θ after adjusting for a. The intuition: if a comparison’s features line up with a judge’s known lean, that judge’s vote on it says almost nothing about humans. So the pair either goes to a human or is spent on purpose to pin the lean down. The older recipe calibrates the judge first and then designs as if it were clean. NAOD puts the bias into the sampling decision itself. Stage four scores it with proxy-policy regret against a reference fit only on the initial data. NAOD reaches 1.99×10⁻³, against 2.81×10⁻³ for a matched target-information design (−29.1%, better on 14 of 15 splits) and 2.45×10⁻³ for D-optimal design (−18.9%, 12 of 15). It also has the best human cross-entropy (0.6699 nats) and the best human choice accuracy (58.27%).

Stage five is where it gets interesting: when does it pay? W is estimated from history, so it carries error, and NAOD is more sensitive to that error than the simpler design. The paper derives a reversal condition. Once the cost of representation error, scaled by the ratio of new to historical samples, exceeds NAOD’s oracle advantage, NAOD loses. In the synthetic check (n = 960 new samples), NAOD wins once the historical sample m is roughly twice n or more. At m/n = 1/4, the plain target-information design beats it. Read that way, the 29% is paid for back in stage one. A deep back catalog of human votes is what lets you estimate each judge’s lean well enough to route around it. An operator with years of logged votes sits far above m/n = 2. A newcomer with a thousand votes sits below the line and should use the simpler design until its history grows. So an old archive does more than train the next reward model. It makes every new human label cheaper, and that advantage compounds. In practice: fit a per-judge deviation from past human–judge overlap, use it to decide which pairs go to paid humans and which to judges, and watch m/n so you know which side of the reversal line you’re on.

The caveats are real. A human choice accuracy of 58.27% is barely above a coin flip, so this is learning a weak signal well, not finding a strong one. The regrets are on the order of 10⁻³, there’s one archive, and the judges are unnamed. Still, the result lines up with a separate scaling-law paper finding that returns to more comparisons grow only like √log M. If more votes buy that little, the leverage is in choosing which votes to collect, and that choice turns out to depend on how many votes you already have.