Paper of the Week #2 — The Score Is in the Abstract, the Bill Is in Table 2

Paper of the Week #2 — The Score Is in the Abstract, the Bill Is in Table 2

Wrapping a coding agent in an outer Planner→Developer→QA loop is this fall's hottest scaling axis. I built the smallest honest version around Claude Code and got the same shape the paper reports: a score gain inside run-to-run noise, at three times the tokens.

Aug 26 – Sep 2, 2026 · Harness-of-Harness · Meta-Harness · StateM · JIT-Agent

The Claim

Harness scaling is real, and it is currently sold without a denominator.

The papers stacking loops around fixed models report genuine score gains, and the headline number is always the score. The cost sits somewhere else: an appendix table, a token count nobody normalizes, or nowhere. That makes "add another loop" impossible to compare against the one control that matters — spending the same tokens on the plain agent. Until harness results are quoted as points per token, a 52% relative gain is not a result; it is a numerator. This week's lead paper is the cleanest example, because it does publish the bill — in a table the abstract never mentions.

The Receipt: HoH, rebuilt small around Claude Code

Yan, Su, Zhang et al. (Shanghai AI Laboratory) · Sep 1 · arXiv:2609.01481

Harness-of-Harness wraps an unmodified coding agent in an outer loop with three roles: a Project Planner that picks a bounded objective, a Developer that implements it, and a QA Tester that evaluates the result against requirements, with artifacts and evidence carried into the next iteration. Nothing inside the agent changes. After three iterations, HoH reports an average relative gain of 52.25% and a maximum of 82.86% over the standalone harnesses (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, Pi with MiniMax-M3) across game development, repo-level SWE, and program reconstruction.

The number the abstract does not carry is in Table 2: cumulative tokens go from 2.59M for one vanilla pass to 8.41M for HoH@3 — 3.25x.

Claude Code is not one of the three agents they wrap, so that is the cell I filled. I wrote a ~120-line harness that runs the same three roles as separate Claude Code invocations — Planner writes PLAN.md, Developer implements it, QA writes its own test file from the spec and reports — with up to two QA→Developer repair cycles. Eight hard Exercism tasks (Forth interpreter, reactive cells, SGF parser, and so on), each graded by the official test suite, which both arms never see. One run per cell, model sonnet.

The single call solved 7 of 8 tasks for $2.04 and 58,498 tokens. The wrap solved 8 of 8 for $5.98 and 177,100 tokens. That is 2.9x the cost and 3.0x the tokens — just under HoH's own 3.25x, measured on a different agent, a different model family, and tasks three orders of magnitude smaller.

Two things did not go the way I expected. First, I assumed the QA→Developer repair loop would be the engine of the whole design. It never fired: on all eight tasks, QA approved the Developer's first attempt, so every wrapped run ended after one QA pass. The entire 3x premium bought planning and independent test-writing, not iteration — and the role breakdown says where it went: QA $2.76, Planner $2.07, Developer $1.15. The implementation was the cheap part. The meetings were expensive.

Second, the score gain dissolved when I touched it. The wrap's one win was rest-api, where the single call chose to accumulate debts between users while the hidden tests expect netting. I reran that task: the single call passed 2 of 3 attempts, the wrap 2 of 2. One run per cell cannot distinguish "the planner catches spec ambiguities" from seed luck, and I am not entitled to claim the former. What survives the reruns is the denominator: the score difference is inside run-to-run noise; the 3x bill is not.

The weaknesses, before the conclusion: these are 15-minute tasks, not HoH's multi-day projects, and outer-loop planning may genuinely need scale to pay off — their FrontierSWE runs go to ten iterations, mine converged in one. My wrap is a skeleton of their design: no warm start, no cross-task evidence archive. One model, one run per cell. And one correction to my own draft of this claim: I began writing "harness papers don't report cost" and had to delete it, because HoH does — in the body, carefully. My complaint is narrower and, I think, fairer: the score is in the abstract, and the bill is not.

The Contrast

Meta-Harness — the counterexample that prices itself

Lee et al. (Stanford) · Mar 2026 · arXiv:2603.28052

If HoH is harness scaling without a denominator, Meta-Harness is the denominator done right. It treats the harness itself as searchable code: an agentic proposer reads the source, scores, and execution traces of every prior candidate and writes the next harness. On online text classification it beats a state-of-the-art context-management system by 7.7 points while using 4x fewer context tokens; one discovered harness lifts 200 IMO-level problems by 4.7 points on average across five held-out models. Better and cheaper, in the same table. This is the reporting standard the field should copy, and it is the strongest evidence that the denominator is an engineering target, not an accounting nuisance.

StateM — the other end of the cost axis, again

Qin, Lu, Wang, Wang · Aug 15 · arXiv:2608.15089

StateM appeared in issue #1 as the operational side of interpretability; it returns here because it is one of the few harness papers that made cost a headline: durable state, recoverable runbooks, and procedural controls lift GPT-5.5 from 83.1% to 92.1% on Terminal-Bench 2.1 for under $15, with 95.3% raw accuracy for GPT-5.6 Sol xhigh across 445 trials at the expensive end. Score and price, both quoted. Between StateM's "$15 run" and HoH's buried 3.25x, the difference is not rigor — both measure — it is what gets promoted to the front page.

JIT-Agent — harnesses become a product, not a craft

Zhang et al. · Aug 26 · arXiv:2608.25593

JIT-Agent trains a model whose only job is to synthesize a task-adapted harness for whatever LLM you hand it, on the fly. Equipped with it, DeepSeek-V4-Flash beats GPT-5.6 by +9.1 on DeepSearchQA and +4.3 on OdysseyBench, and GLM-5.2 gains up to +20.2. I cannot use it as evidence on the cost question — the gains are not normalized per token either — but as direction it completes the picture this week: Self-Harness (June, same lineage as HoH via co-author Hangfan Zhang) made harnesses improve themselves, HoH makes them wrap each other, and JIT-Agent makes them a deliverable. The harness is no longer the part you write by hand.

Where I'd Be Wrong

Two observable results would overturn this issue's verdict. First, the iso-cost control I did not run: give the single agent the wrap's full budget — best-of-three attempts at the same spend — on repo-level tasks rather than katas. If the wrap still wins at equal tokens, my "inside the noise" conclusion is wrong where it matters, and the premium is buying something reruns cannot. I will run exactly this in issue #3 and grade it either way. Second, HoH's own extended runs: if quality keeps climbing through ten iterations while per-token gain stays above the vanilla agent's at matched budget, the denominator complaint fades at long horizons. I will score that when their artifacts permit a matched-budget read.

Ship It · Code of the Week

The one piece of the wrap that earned its tokens is also the cheapest to steal: QA that writes its own tests from the spec, without looking at the implementation's assumptions. In my runs that role produced the sharpest artifact of the week — a QA report that flagged, unprompted, that the spec's example object implies a debt-tracking semantics the instructions never actually state. That is the exact class of bug that survives an agent's own green tests.

Monday morning: before merging agent-written code, add one extra invocation that reads only the requirements and writes an adversarial test file. Do not show it the code's tests. My 120-line harness (hoh_lite.py, in this issue's bundle) does this as its QA role; the curated map of this literature is RUCAIBox/awesome-agent-harness.

The Ledger

Issue #1 promised two things. The matched-loss control ran: both models trained to the promised ≤1.60 validation loss on the same split (ReLU Transformer 1.5608, BDH 1.5458, cosine schedule, seed 1337, sparsity measured at each best-val checkpoint — a protocol change from #1's end-of-training measurement). At matched quality the ordinary Transformer's FFN hidden layer is 97.40% exact zeros against BDH's 77.86% single latent. The falsification condition — ReLU collapsing toward 50% once dead units were ruled out — happened in reverse. #1's claim stands, sharpened: at matched loss, the plain baseline is more sparse. The second promise, running a checked StateM runbook on a real publishing task, happened only as executable pre-publish checks, not the full state machine: partially kept, counted as such.

Reproduction streak: 2 weeks. Next issue: the iso-cost harness control, plus a separate measured post — an edge-native MoE serving stack meeting a datacenter A100, where the install already produced its first finding.

Harness: hoh_lite.py (~120 lines, in the bundle with raw JSON). 8 Exercism tasks, hidden official pytest suites, graded by the runner (agents never see the official tests); model sonnet via Claude Code 2.1.247; one run per cell plus 3 reruns on the contested task. Matched-loss control: matched_loss.py, one A100, 8,000 steps, 4.8B/1.2B activation values counted.

Wall-clock: ~80 min (wrap benchmark) + ~85 min (matched control). Verified on: 2026-09-02.

References

• Yan, Su, Zhang et al., Harness-of-Harness, Shanghai AI Laboratory · Self-Harness (June, same lineage)
• Lee et al., Meta-Harness, Stanford · reference code
• Qin et al., StateM · henryqin1997/statem
• Zhang et al., JIT-Agent
• RUCAIBox/awesome-agent-harness — survey + reading list
• Paper of the Week #1 — Sparse Is Not the Same as Interpretable