The harness improved. The arithmetic did not: a fixed-model repair experiment
A local Qwen3.5 9B harness search improved 36/72 to 50/72 correct attempts, mainly by fixing JSON escaping. We inspect frozen selection, unchanged arithmetic failures, and a full-development evidence ablation.

The winning harness raised correct answers from 36/72 to 50/72 without changing the model. Then we broke the score down by task family. Twelve of the fourteen additional correct attempts came from JSON string escaping. Invoice arithmetic remained at 0/12. The new system prompt called the model a “strict arithmetic engine,” but that description did not make its arithmetic correct.
These are local measurements on 36 unique final-test questions, each run with two seeds at temperature zero. They are not 72 independent questions or a reproduction of Self-Harness's terminal benchmarks. The useful finding is the location of the improvement: an output-contract failure disappeared while a semantic failure survived.
Our executable experiment
We used Ollama 0.40.2 with qwen3.5:9b, a Q4_K_M checkpoint whose metadata reports approximately 9.7 billion parameters. The context limit was 24,576 tokens. The same fixed model solved tasks and proposed changes. We did not use the circulating small-proposer/large-solver setup.
We generated 72 structured tasks with answers computed in Python before inference. Six families covered invoice arithmetic, policy precedence, last-write-wins paid-ledger sums, JSON escaping, extraction from untrusted text, and missing fields that should become null. Each family appeared equally often in development, validation and final test.
The split was 18/18/36. Development failures informed proposals; validation scores selected a configuration; the final 36 questions were scored only after that configuration was frozen. Neither proposer received final-test outcomes. This makes the final comparison different from repeatedly reporting the score used to select a winner.
The editable surface was deliberately small: system instructions, optional JSON extraction, and an optional retry after a schema failure. The answer generator, model weights and splits stayed fixed. A retry could receive expected field names and types, never the correct values. A semantically wrong but well-typed answer would not trigger one.
That distinction matters for a business agent. A parser can certify that total_cents is an integer. It cannot certify that the integer reflects the right discount, tax order and rounding rule. We therefore retained both schema validity and exact value correctness.
The proposer needed a repair before evaluation
The baseline answered 9/18 development questions correctly. Invoice, ledger and escaping were each 0/3; injection resistance, missing fields and policy were each 3/3. We supplied only failures to the main proposer.
Its first four-candidate response contained invalid JSON escaping. We retained that response and added one syntax-only reserialization call, without exposing validation or test answers. A proposal that describes a sensible change still has to become an executable configuration. This failure belongs in the record of an automatic improvement loop.
| Configuration | Validation correct / 18 | Generated tokens |
|---|---|---|
| Baseline | 7 | 246 |
| Candidate 1 | 8 | 169 |
| Candidate 2 | 7 | 183 |
| Candidate 3 | 7 | 215 |
| Candidate 4 | 11 | 188 |
The selection rule maximized correct answers and broke ties by generated tokens; the baseline remained eligible. Candidate 4 combined instructions for invoice calculations, ledger filtering and string preservation. It enabled extraction and schema retries. In the final test it made exactly 72 calls for 72 attempts, so no extra retry was used. The measured improvement did not come from buying additional attempts at the final questions.
The family breakdown changes the interpretation

| Final-test family | Baseline / 12 | Failure-only winner / 12 | Full-development winner / 12 |
|---|---|---|---|
| Escaping | 0 | 12 | 0 |
| Untrusted input | 12 | 12 | 12 |
| Invoice | 0 | 0 | 0 |
| Ledger | 4 | 4 | 4 |
| Missing field | 12 | 12 | 12 |
| Policy | 8 | 10 | 8 |
| Total | 36 | 50 | 36 |
The main winner improved accuracy from 50.0% to 69.4%. That 19.4-point difference depends on our task mixture: escaping accounts for one sixth of the test. A deployment with fewer escaping failures would receive a different aggregate benefit even if every family-level result transferred.
We grouped the two runs of each question before computing a paired bootstrap over 36 task-level differences. Across 4,000 resamples, the 95% interval was approximately 8.3 to 33.3 percentage points. It does not cover uncertainty from generating another batch of proposals, and shared task templates limit generalization beyond this dataset. Treating all 72 attempts as independent would overstate the information in the experiment.
Schema-valid outputs increased from 60/72 to 72/72. Yet 22 of the winner's outputs were valid JSON containing wrong answers. This is a concrete reason to keep format checks separate from task checks. The format gate succeeded completely while the business-answer gate did not.
More evidence did not produce a better candidate in this run
After inspecting the narrow task-specific proposals, but before reading validation or final-test scores, we added a second arm. Its proposer saw all 18 development records, including successes. It generated four candidates under the same editable fields and validation criterion. This was a declared exploratory ablation, not a hypothesis registered before all observations.
Its validation results were 8, 8, 7 and 8 correct. The token tie-break selected the fourth candidate. That prompt concentrated on secure extraction and missing fields, which the baseline already handled well. Its final result was 36/72, with the same family totals as the baseline.
One proposal batch per arm cannot establish that showing successes is harmful. The candidate texts differ, and proposal randomness is not independently replicated. What it does establish is that a larger evidence packet did not guarantee useful repair here. The improvement loop still needs a way to prioritize unresolved failures and preserve existing behavior.

The baseline generated 996 tokens over the 72 attempts, the main winner 752, and the full-development winner 578. These totals exclude proposal-generation cost. All three made 72 final-test calls. The shortest configuration was not more accurate. Output length, schema compliance and task correctness are separate outcomes.
We also avoid a latency claim: the local server was shared, and wall-clock durations included queueing behind other experiments. Generated-token totals are useful workload measurements, not a complete price or throughput comparison.
Connecting this result to Self-Harness
Our experiment changes a bounded configuration around a fixed model. It does not edit arbitrary agent code, run terminal tools or reproduce the paper's larger benchmarks. It gives us a concrete way to read those results: ask what failed, what actually changed, which score selected the change, and what remained broken afterward.
What the model can change
Imagine a coding agent that announces completion without saving the required output file. A harness can ask for the output path earlier, check for the file before stopping, or recover from a failed command. These changes affect the program around the model without changing its weights.
Self-Harness v3, by Hangfan Zhang and colleagues at Shanghai AI Lab, studies this kind of change. A fixed model performs tasks, examines evidence of its failures, and proposes bounded edits to its own harness. The solver and proposer use the same model. The editable surface is declared in advance; the evaluator stays fixed.
We could not match the circulating title “The End of Harness Engineering” and its specific 4B-model/eight-edit description to a primary paper in this source check. This article discusses Self-Harness on its own terms. It does not treat the two descriptions as interchangeable.
The acceptance rule matters
The loop first runs the current harness and collects execution traces. Failed traces are grouped into recurring patterns. The model receives that evidence and proposes small edits aimed at particular failure mechanisms. Candidate harnesses then run against the benchmark again.
The paper separates held-in and held-out tasks. Held-in traces and failure evidence are exposed to the proposer. Held-out tasks are hidden from it and used by the automatic promotion gate.
Algorithm 1 requires a candidate to avoid degrading either split and improve at least one. If several candidates pass, their edits are merged. That is more specific than “keep the single highest-scoring edit.” It also makes the role of held-out scores explicit: they help decide what becomes the next harness.
This approach creates useful review material. An edit has a motivating failure, a limited code surface, and a measured result. An operator can inspect the change and retain a previous version. The model's explanation alone does not authorize promotion.
What the reported gains cover
For Terminal-Bench-2.0, the paper reports the following held-out pass rates. Each row compares the initial and final harness around the same fixed model. We did not rerun these benchmarks.
| Fixed model | Initial harness | Final harness |
|---|---|---|
| MiniMax M2.5 | 40.5% | 61.9% |
| Qwen3.5-35B-A3B | 23.8% | 38.1% |
| GLM-5 | 42.9% | 57.1% |
The experiment uses a fixed 64-task subset of the benchmark's 89 tasks, excluding cases with unstable external web dependencies or unsupported multimodal inputs. Unless otherwise specified, its pass metric averages two attempts per task for each harness candidate. These are conditions attached to the reported gains, not improvements established for every terminal workflow.
The baseline is deliberately minimal: short instructions and basic tools. An extensively tuned production harness is a different starting point. The paper also evaluates SWE-bench Verified and AppWorld, but their task sets and environments differ; combining their percentages would obscure those differences.
Hidden questions still participate in selection
There is a real distinction between showing held-out traces to a proposer and using held-out scores only at a gate. The latter prevents direct access to the questions and their detailed failure evidence. It does not make the final score independent of selection.
This observation does not establish that Self-Harness's improvements are spurious. Useful changes may genuinely transfer. Our simulation isolates a narrower mechanism: choosing the largest noisy estimate can inflate the score used for selection, even when every candidate has identical population accuracy.
We sampled independent binomial success counts for each candidate on a 100-question selection set and a 1,000-question independent test. We used nested candidate counts, so the smaller searches are prefixes of the larger search. Ties select the first candidate. All conditions use the same 60% population success probability.
| Candidate count | Mean selection score | Mean independent test score |
|---|---|---|
| 1 | 59.96% | 59.97% |
| 8 | 66.88% | 60.00% |
| 40 | 70.38% | 59.98% |
| 328 | 73.87% | 59.99% |

For 328 candidates, the average gap was 13.875 percentage points, with a Monte Carlo standard error of 0.033 percentage points. That small standard error describes our estimate within this synthetic model. It does not predict the gap for a real agent.
Real edits are correlated, tasks have different difficulties, and later proposals adapt to earlier evidence. Our simulation includes none of those mechanisms. The count 328 is an illustrative search size, not a claimed Self-Harness run count. The result supports separating selection and final evaluation; it does not quantify the paper's bias.
The operator still designs the process
For an operational version, we would separate failure-analysis tasks, promotion tasks, and a final test opened only after development stops. If the final test then drives another edit, we would record that it has become development feedback too.
We would also specify cost ceilings, permitted code changes, stopping rules, and rollback behavior. A task-success score alone does not capture every operational constraint. These are our design recommendations, not results established by the paper.
Self-Harness offers a concrete way to connect behavioral evidence with executable edits. It may reduce the amount of harness code people need to author directly. People still decide which changes the system may make, how they are accepted, and what evidence will count as a final check.
Reproduction and sources
The download includes task splits and targets, proposed configurations, complete local requests and responses, the frozen selection records, and an offline analysis script. Recomputing the tables from saved results does not require a model. New inference requires Ollama and the matching checkpoint; identical outputs across installations are not guaranteed. The earlier synthetic selection experiment is retained as a separate statistical illustration.
Updated November 16, 2026: added actual local-model harness search, a development-evidence ablation, family-level outcomes and animated visuals.
Subscribe below for research notes with executable evidence and failure cases.
Files for this post
Reproduction package
The harness, raw logs and result CSVs behind this post. No account needed; the README has a command that recomputes the tables without a GPU.
self-harness-evaluation-repro.zip · 4,667 KB