Harness², Stage by Stage

Google Cloud AI Research has open-sourced Harness² (arXiv 2610.14224, Oct 9), a system in which an agent rewrites its own harness on a task it has never seen, “without ground-truth verifiers.” On LAB, a legal-work benchmark, it takes GLM-5 from 59.6% to 71.4% partial-pass in one step and to 78.1% at k=5. Labels and graders are the scarce resource in agent work, so the interesting question isn’t whether it works but where the judgment goes once the grader is removed. The clearest way to answer is to follow a single task through the pipeline. First, a frozen executor (Gemini 3.5 Flash or GLM-5) runs the task under an off-the-shelf harness such as OpenCode or Codex. Then an editor model, Claude Opus 4.8, reads that trajectory plus a library of past harnesses and writes two candidates. One is an invariant edit (planning, tool use, context, delegation) and the other a domain edit (the field’s procedures), and either can touch any of eight files, from the system prompt and guardrails to skills, scripts and plugins.

Each candidate then runs once on the same task. The authors are explicit that the output of this probe is a “behavioral contrast” against the reference run, meant to inform the next step “rather than verify task correctness.” In the Compose step the editor reads the reference, both candidates, their diffs and the contrasts. It then merges, revises, adopts one, or keeps the current harness, and every decision has to cite a specific observation. The successor runs and its deliverable is submitted. That is four full executions and two editor sessions for one step, and about sixteen executions at k=5. For k>1 there is one more stage: an “agentic selector compares anonymized deliverables against the task request and source materials,” with no scores. This is where the missing grader went. It has become a rubric-less pairwise judge. A final library stage stores a lesson only when an execution contrast supports it, so merely picking a winner doesn’t count as evidence. Removing the library costs 3.1 points on WorkBuddy and 1.8 on JobBench.

The gains are real and rise steadily with k. GLM-5 on OpenCode goes from 64.2 to 75.7 on WorkBuddy, and to 79.6 at k=5. On WB-Code it goes from 58.4 to 77.1, which puts a tuned GLM-5 above base Gemini (72.4). Using the same model as both editor and executor still helps: Opus 5 gains 3.3, 5.5 and 3.0 points. No fixed production harness wins everywhere (Claude Code leads on Code and Office, Codex on Web), and Harness² started from OpenCode beats the best of them by 4.6 to 5.8 points. The failure appendix (E.6) shows where the process breaks. Its nine regressions cluster at the last stage: on WB-Code the selector chose a deliverable scoring 55 over one scoring 95, and on LAB a narrower output dropped a score from 54 to 26. A second failure sits upstream. When the same wrong premise, such as a literal-key requirement or an unchecked cache assumption, appears in both candidates, contrasting them can’t expose it, and that cost 2 to 11 points. One external grader, JobBench’s, is also Opus 4.8, the same model as the editor, and the paper itself mentions “shared model biases.”

Three practical things follow from walking the stages. First, the selector is a router with a judge inside it. Its worst error is a pairwise-preference error, so it should be calibrated on human pairwise preference data before anyone spends more on probes. Second, the paper places Harness² on a token Pareto frontier (Figure 6) but prints no token or dollar figures, so the deciding number is still missing: cost per finished task, compared against simply calling a stronger model once at the same spend. Third, the two candidates come from one editor and share its blind spots, so a probe written by a different model family is the cheap fix for premises that propagate across both. “No grader” turns out to mean the grading has moved somewhere cheaper and uncalibrated. Pairwise judging of that kind is something the field already knows how to measure.