Models & Algorithms•SOTAAZ Lab••KR

Does a Code Reviewer Help a Local Coding Agent on an 8-12 GB GPU? Qwen3.5, Qwen3 and gpt-oss-20b Reviewing Themselves

Same-model review at the agent's 'done' raised hidden-test scores by at most 0.031 on four Python tasks, and a content-free 'keep going' did as well. The reviewer found real bugs; the agents mostly didn't fix them. A100, 8 and 12 GB budgets.

Does a Code Reviewer Help a Local Coding Agent on an 8-12 GB GPU? Qwen3.5, Qwen3 and gpt-oss-20b Reviewing Themselves

Does a Code Reviewer Help a Local Coding Agent on an 8-12 GB GPU? Qwen3.5, Qwen3 and gpt-oss-20b Reviewing Themselves

When a coding agent says it is done, a second model can check the work before you accept it. In October 2026, researchers at Cornell and Alibaba reported in "Rethinking Verification in Agent Harnesses" that such a reviewer raised coding benchmark scores for seven frontier coding models. We tried it on the small models that fit an 8 or 12 GB card. There, the only reviewer you can afford is usually the agent's own model, because the card has no room for a second one.

Our score is the share of hidden tests the agent's code passes: tests the agent never sees, 1.0 for all of them and 0 for none. On four Python tasks, the reviewer raised that score by at most 0.031 when it checked the agent's "done", about 3 more hidden tests passed in 100. A message that didn't look at the code at all, the same "not finished, keep working" every time, raised it at least as much.

The reviewer itself was not the weak part. Qwen3-14B's reviewer wrote this about a todo app the agent had declared finished:

The --due argument is not properly validated. [...] The list --overdue filter is not implemented.

Both were true. The agent read the feedback, ran pytest once, changed no file, and replied:

The due date functionality is now fully compliant with all requirements.

The hidden tests still failed 4 of 6. Of the 47 times a reviewer correctly objected, the score went up 13 times.

The conclusion

These results are for four small Python tasks, a 16K context, Q4_K_M models fully on the GPU, and one reviewer pass. All runs were on an A100 with each card's memory as a budget.

  • Reviewing at "done" barely moves the score. Across five models, a same-model reviewer raised the model score by +0.000 to +0.031. None of those gains is clear of run-to-run noise; Qwen3.5-9B's +0.031 comes closest.
  • What the reviewer pointed out didn't decide the score. The reviewer reads the code and names specific problems, such as "--due is not validated". For comparison we also sent the same sentence every time, "not complete, keep working", without looking at the code. It matched or beat the reviewer at "done" for all five models.
  • The reviewer finds problems; the agent mostly doesn't fix them. In 14 of 47 correct objections the agent changed nothing.
  • It also misses. It approved 14 of the 37 "done" stops that still failed hidden tests.
  • False alarms were rare. It objected to 3 of 18 runs that already passed every hidden test.
  • Reviewing also when the agent runs out of turns clearly helped two models. Qwen3.5-9B went up 0.042 and Qwen3-14B 0.070. A plain "keep going" there gave Qwen3.5-9B 0.075.

What a reviewer does, and what the paper found

A coding agent works in a loop: the model reads files, edits them, runs tests, and at some point replies without a tool call. That reply is its "done". A reviewer, which the paper calls a critic, steps in at that moment. It is a separate model call with a fresh context. It gets the task and the changes, can read files and run commands, and returns a verdict: done, or not done with a list of problems. If not done, the list goes back to the agent as a user message and the agent continues.

Zhang et al. (Cornell, Alibaba), "Rethinking Verification in Agent Harnesses", measured this with seven frontier coding models through a cloud API, all reviewed by Qwen3.7-Plus. The critic raised Terminal-Bench by 5.9 to 9.5 points on average and SWE-bench by 3.9 to 6.6. The paper also tested a rejection that says nothing about the work, which raised Terminal-Bench by 6.0 and 6.2 points.

The paper's critic is a different model from the agent. On one 8 GB card you can't load a second model next to a 9B agent, so the question becomes: can the model review itself?

How we tested

We used the five models that fit 8 or 12 GB in Part 3 of this series, all at Q4_K_M, with the same agent, server flags and tasks:

  • 8 GB: Qwen3.5-4B, Qwen3.5-9B, Qwen3-8B
  • 12 GB: Qwen3-14B, gpt-oss-20b

The four tasks are small Python repos with visible tests and separate hidden tests: add due dates to a todo app (T1), fix a bug whose traceback points the wrong way (T2), add stock reservations across two files (T3), and write a version-number parser from a spec (T4). A run's score is hidden tests passed divided by hidden total; a model's score is the mean over its runs.

Each model ran each task five times, 100 runs. We stopped every run at the first of two points: the agent said done (with the visible tests passing), or it used all 30 turns. We saved the files and conversation there and scored the saved files. That score is the baseline. Then we continued from the same saved state three ways, each with up to 15 more agent turns:

  • Keep going. One fixed user message, taken word for word from the paper: "An automated verification pass has determined this task is NOT complete. Keep working; do not finish yet." It says nothing about the work, so it measures what extra turns alone do.
  • Self-review. The same model on the same server, in a fresh context, as the reviewer. We used the paper's critic prompt word for word. The reviewer saw the task and a diff and could list files, read them and run commands in a throwaway copy, for at most 15 turns. The hidden tests were in neither copy. An approval ended the run at the baseline. An objection went back to the agent, and whatever it did next was final, as in the paper.
  • Cross-review, for the two Qwen3.5 models only: 4B reviewed by 9B and 9B by 4B, both loaded at once.

Code that the hidden tests could not even import scored 0, as in Part 3: 10 baselines and 18 continuations. Seven continuations stopped on a server error, five of them by passing the 16K context, and we scored their files as they were. One gpt-oss-20b run hit a server error before its stop, so it has no continuations and keeps its baseline in every table.

Because every branch starts from the same saved state, the comparison isn't blurred by differences earlier in the run. The same model on the same task scores a little differently from run to run, so we checked whether a gain could be luck. We redrew each task's five runs at random and averaged, 20,000 times (seed 20261113). We call a gain clear only if it stayed above zero in at least 97.5% of those redraws.

Reviewing at "done" barely moves the score

In a real agent, a reviewer at "done" only fires when the agent says done. So the fair number is the model score over all 20 runs, with the reviewer's result replacing the baseline where the agent said done and the baseline kept elsewhere.

Grouped bars for five models: change in model score when the reviewer or a plain 'keep going' runs at the agent's 'done'. Self-review: Qwen3.5-4B +0.012, Qwen3.5-9B +0.031, Qwen3-8B +0.008, Qwen3-14B +0.026, gpt-oss-20b +0.000. Keep going: +0.012, +0.033, +0.008, +0.038, +0.006
ModelSaid done (of 20)BaselineSelf-review at "done""Keep going" at "done"
Qwen3.5-4B40.678+0.012+0.012
Qwen3.5-9B110.639+0.031+0.033
Qwen3-8B90.178+0.008+0.008
Qwen3-14B140.459+0.026+0.038
gpt-oss-20b60.773+0.000+0.006

The largest gain is Qwen3.5-9B's +0.031, which stayed above zero in 97.3% of resamples, just short of our 97.5% line. Its "keep going" gain, +0.033, sits at 97.4%. Part 3 left Qwen3-14B the most room, since it said done 11 times in 12 and was still failing some hidden tests in 10 of those. Here its reviewer added +0.026.

Most runs did not change at all. Of 99 self-review runs, the score went up in 13, down in 7 and stayed the same in 79.

(Baselines here are 20 new runs per model, not Part 3's 12. Qwen3.5-9B scored 0.639 this time against 0.781 there, and gpt-oss-20b 0.773 against 0.750.)

The reviewer finds problems; the agent mostly doesn't fix them

The score barely moved, but the reviewer was good at spotting problems. To check that, we compared each verdict with the hidden tests:

  • If the code failed any hidden test, it still needs fixing, and the right verdict is "not done".
  • If it passed all of them, nothing needs fixing, and the right verdict is "done".

Here are all 99 self-reviews sorted that way:

Reviewer saidFailed some hidden tests (needs fixing)Passed all hidden tests
"Not done"47 (right)3 (false alarm)
"Done"20 (missed)12 (right)
No verdict143

59 of its 82 verdicts were right, including "not done" on 47 runs that needed work. The agent then:

A horizontal bar splitting 47 correct objections: 14 with no file changed, 14 changed but same score, 13 higher score (5 of them passed every hidden test), 6 lower score
  • changed no file in 14, like the todo app above,
  • changed files without moving the score in 14,
  • raised the score in 13, passing every hidden test in 5,
  • lowered the score in 6.

When it worked, it looked like this. On the bug-fix task, Qwen3.5-9B had fixed the reported bug and stopped at 6 of 8 hidden tests. Its reviewer ran the code on a file with comment lines, saw it crash with IndexError, and wrote back that the agent "did NOT implement the README requirement to skip blank lines and lines starting with '#'". The agent added the check and reached 8 of 8. Qwen3-14B's reviewer caught a similar gap in the stock task from the diff alone, in one turn: old stock files without the new reserved key raised KeyError, and the fix took the run from 7 of 8 to 8 of 8.

Qwen3-8B, the weakest model, objected the most: in 18 of its 20 runs. Its edits fail often, as Part 3 found, and the objections rarely turned into points (+0.008 at "done").

Where the reviewer was wrong

Misses were common. Of the 37 times the agent said done while still failing some hidden tests, the reviewer approved 14, and an approval ends the run where it is. Qwen3.5-9B approved 4 of its 7 such stops.

False alarms were rare. Of 18 runs that already passed every hidden test, the reviewer objected to 3. The reviewer gave no usable verdict in 17 of 99 runs, even after we asked for one at its 15-turn limit. Qwen3.5-4B, the smallest model, accounts for 11 of those 17.

Revision rarely made things worse. 7 of 99 self-review runs ended below the baseline, against 9 of 99 for "keep going".

When the agent ran out of turns

More than half of the runs (55 of 100) used all 30 turns without saying done. A reviewer at "done" never sees those. Running the same step there too changed the picture for two models:

ModelSelf-review, both stops"Keep going", both stops
Qwen3.5-4B+0.007−0.043
Qwen3.5-9B+0.042+0.075
Qwen3-8B+0.045+0.076
Qwen3-14B+0.070+0.050
gpt-oss-20b−0.013−0.019

Bold gains are clear of noise by our 97.5% line. At a turn-limit stop, "keep going" is simply 15 more turns. It helped Qwen3.5-9B and hurt Qwen3.5-4B, which lost 0.056 at those stops.

Two models on a 12 GB card

Cross-review needs both models loaded. Qwen3.5-4B and 9B together peaked at 9,816 MiB. That is over an 8 GB card's budget of 7,664 MiB and under 12 GB's 11,760, as in Part 3, where the budget is the card minus 512 MiB for the desktop and a 16 MiB margin. Each process here includes the A100's CUDA context, so a consumer card may need a little less.

With 9B reviewing it, 4B gained +0.037 at "done", the one clear gain at "done" in this test (98.9% of resamples). That comes from 4B's four "done" stops, three of which went up. 9B reviewed by 4B gained nothing at "done" (+0.000) and +0.015 over both stops, less than with its own reviewer.

What it costs

A self-review run added a median 0.66 times the original run's wall time, from 0.50 for gpt-oss-20b to 0.82 for Qwen3.5-4B. "Keep going" added 0.50, and cross-review 0.81. Time numbers on the A100 may vary somewhat with server conditions; the scores, token counts and memory do not.

A same-model reviewer also generated a median 4,100 to 11,300 tokens per run, 37% to 74% of what the original run generated. A self-review needs no extra memory, since it uses the same server.

What to try on your card

If you run Qwen3.5-9B on an 8 GB card and want something after the agent stops, start with the cheaper step. Send one fixed "not finished, keep working" message and allow 15 more turns. That gained 0.075 on these tasks, more than a self-review (+0.042), in under half the time (0.32 of the original run against 0.77). If you do add a reviewer, check whether the agent actually changed a file after an objection. In 14 of 47 correct objections it didn't, and a git diff after the revision shows that without running anything.

Part 3, which model to run on 8, 12, 16 and 24 GB, has the scores and memory behind these five models. Subscribe below to see the next test in this series.