Harness²: A Loop That Edits the Harness Without an Answer Key

Harness²: A Loop That Edits the Harness Without an Answer Key

We ran Google's Harness² demo on this machine because it needs no model credentials: two ledger tasks run with k=2, which produced 28 executions and 32 editor, composer, selector and reflector session calls, and a base harness built to sum only the positive amounts, so the first task printed Total: 105.25 where the signed sum was Total: 85.15. The paper behind it (arXiv:2610.14224) is about something this site has been circling for a while: the harness around a model — prompts, rules, memory, tools, skills — as the object of optimization, instead of the model. One Propose–Probe–Compose step took GLM-5 under OpenCode from 64.2 to 75.7 points on WorkBuddy-Bench, and raised JobBench by 4.5 points under Gemini 3.5 Flash. Both are k=1 results, which skip the final selection step; the selector only runs when k is 2 or more. The part worth reading carefully is not the gain but the seam: the loop picks its next harness without ever seeing a correct answer, and the paper's own failure analysis shows that step choosing a weaker candidate. This post records what we read and what the demo showed. What that selection step actually looks at when it picks, we follow through the released code in the next post.

The terms in those numbers:

A harness, and its eight editable parts

The repository README defines the term: "A harness is the system around a model that enables it to act as an agent: prompts, tools, memory, skills, orchestration, and more." Harnessing is the process of optimizing that system; the paper's own wording is optimizing the runtime scaffolding around a frozen base model. The eight editable parts the OpenCode substrate exposes (system prompt, guardrails, memory, tool guidance, subagents, skills, scripts and plugins), and where the phrase "harness engineering" comes from, are covered in Part 1 of this series. One detail matters here: on OpenCode the code caps the always-on text (system prompt, guardrails, memory and tool guidance) at 24,000 characters combined. The source gives the reason as "every token here competes with the agent's own output budget".

Two recursions, one of them without an answer key

Task-level recursion refines the harness for the task at hand. A harness editor proposes edits, the candidates are executed against the same task, and a composer builds a new harness from the behavioral contrast between those executions. Nothing in that loop sees a correct answer. That is the paper's central problem statement: open-world tasks are novel at test time, so there is no verifier to steer the edit, and the editor has to read the difference between two runs instead.

Harnessing-level recursion is the second, slower loop. After a task finishes, the adopted harness, together with the process and lessons that justified it, goes into a per-domain library, one store for each domain such as coding, web or office. The next task in that domain reads the library before it starts.

The paper shows that the library is used with two numbers. The first is overlap: for each modified file, it counts how many lines already appear in a library entry. Some overlap exists in 22% of modified files on LAB and WorkBuddy-office and 47% on JobBench, so library content does flow into new harnesses. The second is how much is copied: only 2-10% of modified files keep at least half their lines from an entry. The authors read that as adaptation being more common than copying. The rest of the modified files (78% on LAB and WorkBuddy-office, 53% on JobBench) shared no lines with an entry. They also counted where edits land, and skills and scripts account for roughly half.

The two loops are what separate Harness² from a one-off prompt edit: the first loop makes a harness better for one task, the second makes the next task cheaper to harness.

The reported numbers

Table 1 and Table 2 of the paper report one improvement step against the base harness, averaged over three runs.

The parentheses are standard deviations over three runs. The +11.5 is far larger than the two deviations on the average (1.4 and 2.2), but the per-domain base deviations reach 5.2 on WorkBuddy-code and 6.9 on web. Spending more recursion budget adds more gain: in the same GLM-5 configuration, k=1 takes WorkBuddy-code from 58.4 to 77.1 and web from 59.2 to 69.1, and k=5 adds 4.8 on code and 5.3 on web (3.9 on the three-domain average) on top of that.

Two comparison numbers matter more than the headline, and first a word on what is being compared. The paper runs Harness² against other methods under the same conditions: the same frozen model, the same starting harness, the same task stream and the same judge. There are three kinds of competitor. ReasoningBank never edits the harness; it prepends memory notes distilled from the most similar past task to the task prompt. Harness Scaling runs a per-task revision loop on the harness but has no library. Meta-Harness optimizes a harness offline on separate training tasks and deploys it unchanged at test time.

The first number is that these competitors do not all help: with GLM-5 under Codex, ReasoningBank scores 50.2 on coding tasks against 56.8 for the base harness, 6.6 points lower. The second is the dagger on Meta-Harness. In the WorkBuddy-Bench comparison with Gemini 3.5 Flash under OpenCode it is the strongest baseline, but it received separate training tasks to optimize on before testing, while Harness² learns only by working through the test tasks in order. The starting lines differ, so the two should not be read as a like-for-like contest. A number points the other way too: on LAB partial pass rate, ReasoningBank with GLM-5 under OpenCode reaches 71.3 (1.3) against Harness²'s 71.4 (2.9). The +11.8 pp above is measured against the 59.6 base, so the size of the gain depends on what it is compared with.

Two protocol details decide how to read the scores. One is run-to-run wobble: the paper shuffles the task order in each of the three runs, and the repository's experiment notes (docs/EXPERIMENTS.md) say the seed does not control model sampling, so re-running the same setup gives slightly different scores, and the parentheses are that spread. The other is grading: each score includes parts judged by an LLM, so the yardstick is not perfectly fixed either. Appendix F names the cost of recursive refinement as a limitation and says the trade-off between spending and gain is characterized only over a bounded budget range.

Which parts carry the gain

The paper removes component groups from the already-improved harness and re-evaluates it without rerunning the improvement (Appendix E.1, WorkBuddy-Bench, Gemini 3.5 Flash + OpenCode, k=5).

Skills cost the most when they are removed, then scripts, then the always-on prompt text. Every harness with one group removed still beats the base harness, which says the gain is not carried by one component. The authors are careful not to claim more than that: the ablations "support contributions from multiple component groups, but do not establish additive effects."

We ran the released demo

The repository ships a deterministic local demo that needs no model credentials, so we ran it on this machine to see the machinery rather than the claim.

The demo's base harness contains a bug planted for the demonstration. Its scripts/summarize.py sums only positive amounts, so the first task (100.25, −20.10, 5.00) comes out as Total: 105.25 when the signed sum is Total: 85.15. The editor's edit removes the if amount > 0 clause, and the library entry records it as "dropped the positive-only filter from the amount sum". The probe then reports Total: 85.15 against the base run's Total: 105.25, and the selector keeps the candidate whose output matches the ledger. The second task (35.50, −10.25, −5.00) behaves the same way, Total: 35.50 before and Total: 20.25 after.

What the run also shows is the cost shape. With two tasks and k=2 across both recursion modes, the pipeline made 28 executions and 32 editor, composer, selector and reflector session calls. Per task that is 8 executions in parallel mode (two base references, two invariant candidates, two domain candidates, two compositions) and 6 in sequential mode, where the base references are reused. Together that is 14 executions per task, so 28 for two tasks. The paper also reports agent steps (#Steps) and, in Figure 6, the total compute including the tokens spent on improvement; this small run shows how that compute accumulates.

The library entry is the artifact we found most interesting, and it is a small Markdown file (excerpted below, formatting tidied): the adopted harness, the reason it was chosen, what each edit did, and a transferable lesson written as a cue, a lesson, an evidence line and a confidence.

- adopted: `agg1`  (run `demo-oc-agg2/ledger-1/agg1`)
- components: 1 scripts; 62 always-on chars
...
- cue: When a ledger includes credits and debits
- lesson: Include both signs when summing in scripts/summarize.py.
- evidence: The scripts/summarize.py row -- removing the positive-only filter changed
  the probe total to include the credits omitted by the base trajectory.
- confidence: high for this observed arithmetic contrast

One caveat has to sit next to those numbers: in the shipped demo the editor and the selector are scripted, not model-backed. The run demonstrates the mechanism and the artifact format. It says nothing about whether a model-driven editor finds edits as good as the scripted one. The same goes for selection: the demo's selector computes the ledger total itself and picks the matching candidate, so the step that must choose without an answer key cannot be seen in this demo.

Where the loop can pick wrong

Appendix E.6 groups nine regressions into three kinds, and the first kind is the one to keep in mind: selection can prefer a weaker candidate, 55 chosen over 95. On WorkBuddy-code, a retrospective evaluation shows an output scoring 55 chosen over one scoring 95. On LAB, narrower coverage takes the score from 54 to 26. The second kind is an incorrect premise propagating across candidates, costing 2 to 11 points. The third is outcome variability and missing judge verdicts, which makes attribution hard.

The limitations section is explicit about why this happens: without ground truth, refinement "relies on behavioral contrasts and model judgment", which "remain sensitive to execution variability and shared model biases" and "do not guarantee correctness". The same caution applies to the reuse numbers. Re-running a task with its own library entry removed scored 79.4 against 77.5 on WorkBuddy-Bench (a different setting from the table above: 77.5 is the Gemini 3.5 Flash with OpenCode k=1 score) and 37.7 against 36.0 on JobBench. The baseline is the original k=1 online run, the re-run can also use entries from later tasks, and the authors note that repeated execution may contribute. So it cannot be read as removing an entry raising the score, and they call it an offline reuse analysis rather than an estimate of online gains.

Transfer across models is real but uneven, which is the other half of the same caution. A harness refined for Gemini 3.5 Flash raises GLM-5's WorkBuddy-code score from 58.5 to 75.8 (the base value of the cross-model experiment, listed separately from the 58.4 in the earlier table), and the same harness adds 0.3 points on Gemini 3.7 Flash's code subset and 2.6 points for Claude Opus 4.8. The authors write that all three models improve on both code and office, with office gains of 6.4 for Gemini 3.7 Flash and 1.1 for Opus 4.8.

What the next post looks at

The part of Harness² worth a close look is the selection step: when the loop picks a candidate with no answer key, what does it base the choice on? The paper reports failures of that step as cases but gives no hit rate (searching the text for oracle, regret and selection accuracy gives zero results). The demo here scripts the selector, so that step stays out of view. Part 4 reads the released code to see what the selector takes as input and what it uses to choose.

Harness² makes the harness the object of optimization, and once the budget allows two or more candidates a step must choose without an answer key; in the paper's own failure analysis it chose a 55 over a 95.

Method note

• Paper: Harness²: Recursive Agent Harnessing for an Open World, arXiv:2610.14224 (submitted 9 October 2026), Ruiyao Xu et al. (led by Google Cloud AI Research, with Northwestern University). Numbers quoted above are from Tables 1–2 and Appendices E.1–E.3, E.6 and F, checked against the arXiv HTML.
• Code: google-research/harness2, Apache 2.0, commit 1cfe254, package version 0.1.0. The README states this is not an officially supported Google product.
• Demo run: uv venv --python 3.12 .venv, uv pip install -e '.[render]', harness2-demo --output runs/demo --k 2, on Linux, Python 3.12, 20 October 2026. No model credentials were used and no model was called.

References

• Harness²: Recursive Agent Harnessing for an Open World — arXiv:2610.14224
• Code, demo and benchmark guides — github.com/google-research/harness2
• Project page and case videos — harness2.github.io