Which Local Coding Model Should You Run on Your GPU? 8, 12, 16 and 24 GB on 4 Python Tasks: Qwen3.5-9B, gpt-oss-20b, Flint, Qwen3.5-27B

Which Local Coding Model Should You Run on Your GPU? 8, 12, 16 and 24 GB on 4 Python Tasks: Qwen3.5-9B, gpt-oss-20b, Flint, Qwen3.5-27B#

Which local model should a coding agent use on an 8, 12, 16 or 24 GB card? We ran nine models through four coding tasks, three times each, 108 runs in all. We scored the resulting code on tests the model never saw: passing all of them scores 1.0 and passing none scores 0. Some of those tests pass on the original code, so changing nothing at all still scores 0.115. All runs and memory measurements were on an A100, assuming each card's capacity. 8 GB matches the RTX 4060, 12 GB the RTX 3060 12GB, 16 GB the RTX 4070 Ti Super, and 24 GB the RTX 4090.

The conclusion#

These answers are for four small Python tasks, a 16K context, and models fully on the GPU. Every model but Flint was tested only at Q4_K_M.

• 8 GB and 12 GB: use Qwen3.5-9B Q4_K_M. Going from 8 to 12 GB added no clearly better model. Qwen3-14B, which needs 12 GB, scored 0.485, and gpt-oss-20b (0.750) can't be separated from 9B on three runs per task and leaves only 46 MiB. We did not measure a higher quant of 9B that would fit 12 GB.
• 16 GB: Flint UD-Q2_K_XL is on top at 1.000. We built our agent and one task with it, though, so it did not start on equal terms. Without Flint, 9B and gpt-oss-20b can't be separated, and both fit 16 GB with room to spare.
• 24 GB: either Flint or Qwen3.5-27B (0.953). These tasks can't separate them. Flint carries the same caveat as at 16 GB; 27B is the pick without it.
• Don't use Qwen3-8B with this agent. Even if it fits 8 GB, 128 of its 179 code edits failed.

By card. Memory left is the budget (card size minus 512 MiB for the desktop and a 16 MiB margin) minus the agent's peak.

The 11 runs where the model stopped by itself and still scored below 0.5 were all Qwen3-8B or Qwen3-14B. With other models too, it is safer to check the changed code yourself.

How we tested#

Each model ran on llama-server (build 1137) with this command, fully on the GPU:

llama-server -m MODEL.gguf -c 16384 -ngl 99 -fa on --parallel 1 --jinja

The agent is the agent.py from Part 3 of our local agent series, at temperature 0.6, with a limit of 30 model turns and a 30-minute timeout. Every permission prompt was answered yes, and no decision model ran next to it. Every model got the same 16K context, so the scores compare models and not context.

There were four small Python tasks. Each has tests the model can see and separate hidden tests:

• T1: add due dates to a todo app (6 hidden tests). The visible tests already pass.
• T2: fix a bug whose traceback points the wrong way (8).
• T3: add stock reservations across two files, where the visible tests already pass before you touch anything (8).
• T4: write a version-number parser from a spec (10).

A run's score is hidden tests passed divided by hidden total, using whatever the files held when the run ended. A model's score is the mean of its 12 runs.

Four of the 108 runs ended with code that could not be imported, and we counted them as 0: two for Qwen3-8B, one for Qwen3-14B and one for gpt-oss-20b, all at the 30-turn limit. Qwen3-14B's used syntax valid only from Python 3.12, and we scored on 3.11.4. We did not check the cause of the other three.

What fits which card#

For memory, we started each model and read the process's number from nvidia-smi, then tracked the peak during agent runs. Card sizes are in binary units: 8 GB is 8,192 MiB, and after the desktop share and margin the budget is 7,664 MiB.

Treat the amber cells as undecided. The 1,000 MiB line is a cut we chose. Your desktop may use more than 512 MiB, and the roughly 480 MiB that no log line lists in Part 1 of this GPU series, which looks like the CUDA context though we did not measure that directly, may differ on your card.

This post's budget is also more generous than the one in that post, which counted an 8 GB card as 8 × 10⁹ bytes, took off 512 MiB and got 7,117 MiB. Under that stricter count, exactly the three amber cells stop fitting: Qwen3-8B at 8 GB, Qwen3-14B and gpt-oss-20b at 12 GB. The top model on each card does not change, because none of the three is on top. Your own number comes from nvidia-smi --query-gpu=memory.free --format=csv.

Two others narrowly missed the 16 GB budget: Devstral Small 2 24B by 760 MiB at peak, Qwen3.5-27B by 1,260. An 8-bit KV cache (--cache-type-k q8_0 --cache-type-v q8_0) brings Devstral's loaded number down to 15,394 MiB, which leaves 462 MiB on this post's budget and does not fit the stricter one above. We ran no agent in that setup, so it is not in the 16 GB ranking.

The scores#

T1 to T4 are per-task scores. "Without T1" drops T1, the task we built with Flint.

Speed is the median over each model's runs on an A100 shared with other jobs. It is not what your card will do.

Three runs per task is few, so we resampled the runs inside each task 20,000 times and asked how often one model stayed above another.

• Clearly apart: 9B over 4B, Flint over 9B and gpt-oss-20b, and 27B over 9B stayed on top in every resample or nearly so.
• A tie: 9B stayed above gpt-oss-20b in only 61% of resamples. Drop T1 and the two swap places.
• Can't say: Flint and 27B. Flint passed everything, so a resample can't move it, and shuffling model labels inside each task produces a gap this large 10% of the time (one-sided permutation test, p = 0.10). With three runs per task, this data can't give a smaller value. The tasks ran out of room at the top.

Flint's row needs its own warning. We built T1 and agent.py with Flint in the local agent series, so it was never a neutral entry, and we listed it as a reference row from the start. It is also the only model whose server log says its chat template preserves reasoning between turns. We did not measure what that does. And "Flint" here always means this one UD-Q2_K_XL file. We have no Q4 row for it, and we measured no lower quant of 27B that might fit 16 GB.

T4, the version parser from a spec, was the hardest task at a mean of 0.504. We had predicted T3.

What happened to Qwen3-8B#

Before any run, we wrote down the score we expected from each model. For Qwen3-8B, an older model that just fits an 8 GB card, we wrote 0.45: nearly half the tests passed. It scored 0.176, not far above the 0.115 for changing nothing.

Most of its attempts to change the code failed, far more than any other model's. In one run, 27 failed in a row.

Edits include write_file. A turn can hold several failed edits, so a run with 5 in a row can show as just a few orange squares.

An edit in agent.py replaces a piece of old text with new text, and the old text must appear exactly once in the file. To see what the model was sending, we reran Qwen3-8B on T2 and T3 afterward with full tool arguments in the log. These six runs are not part of the score. They passed 19 of 48 hidden tests, against 10 of 48 on the same tasks in the scored round, so this does not happen in every run: two of the three T2 reruns reached 6 of 8. In the other T2 run, all 25 edits failed, and 23 of them repeated the previous attempt's old text exactly. That text starts like this:

def parse_line(line):
    if line.strip().startswith('#') or not line.strip():

The real function has no such second line, and this run never opened orders.py with read_file; it read the README and report.py and ran pytest. Every time, the agent answered old appears 0 times in orders.py, it must appear exactly once, and the model sent the same text again until turn 30.

Why it does this, we don't know. The cause could be the chat template, how the model reasons, or the model itself; we did not log its reasoning. agent.py clears old tool output when the context passes half full, but that alone is a weak explanation: 9B went through 54 clearings against Qwen3-8B's 40. This result says this GGUF did badly with this agent's edit tool. It says nothing about Qwen3-8B with another agent.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

The model that said it was done#

Part 3 of our local agent series showed an agent reporting success on code nobody had changed. It happened here too, though this time nothing was cut off and the model wrote the claim itself. Qwen3-14B stopped after one pytest on T1 in all three runs, with no edits, because T1's visible tests already pass. In the first run it then wrote:

All tests passed with pytest, confirming the implementation is correct.

The same summary listed "Added comprehensive tests in test_todo.py", which it had not done. All three runs scored 2 of 6, the same as the untouched code.

Of the 58 runs where the model stopped by itself and the visible tests passed, 11 scored below 0.5. All 11 were Qwen3-8B or Qwen3-14B, and 4 of them had no edits. So a passing pytest at the end says little until you check that anything changed.

The other 50 runs hit the 30-turn limit. They averaged 0.594, against 0.796 for runs that stopped by themselves, but the limit is neither pass nor fail: 12 of those 50 had already passed all hidden tests when it arrived.

What we predicted#

We did predict 9B at both 8 and 12 GB, so that result did not surprise us. Our largest misses were Flint (0.60 against 1.000), Qwen3-8B (0.45 against 0.176) and 4B (0.35 against 0.610).

Trying the 8 GB pick#

hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-Q4_K_M.gguf --revision 3885219 --local-dir models
llama-server -m models/Qwen3.5-9B-Q4_K_M.gguf -c 16384 -ngl 99 -fa on --parallel 1 --jinja --port 8080

Part 1 of our local agent series builds the agent that talks to this server. Before you trust a smaller card's number, check nvidia-smi for the process's memory during a real task.

Limits#

• Memory numbers. Measured on an A100 and compared with each card's capacity, so yours may differ a little. Check tight fits yourself, such as gpt-oss-20b with 46 MiB left.
• Q4_K_M only apart from Flint, and the label is loose. A 12 GB card could also hold a higher quant of 9B, which we did not measure, so the conclusion holds only among the files we tested. Most of gpt-oss-20b's quantized tensors are MXFP4 and Q5_0 despite its name. Flint's "Q2" is a mix that includes Q5_K.
• Four small Python tasks. The ranking is for these tasks. Flint passed everything and 27B nearly everything, so these tasks can't tell the top models apart, and harder tasks may reorder them.
• Three runs per cell. Per-task numbers move in steps of a whole run.
• Full GPU only. We did not try CPU offload, which is how many people run a bigger model on a smaller card.
• 16K context for everyone. A longer context could change scores and memory. We did not test it.
• Turns, not tool calls. The 30-turn limit favors models that send several calls in one turn. Flint had 29 such turns; gpt-oss-20b and Qwen3-Coder-30B had none.
• Default templates and thinking. We changed neither per model, and a tuned setup could score differently.

Next step#

To check a model against your own card before you download more, use two short runs and a straight line. The method gave exactly the measured 16K number on all nine models here.

We also measured whether Idol, which turns one photo into a 3D model, fits an 8 GB card. On an A100 capped at 8 GB it peaked at 7,380 MiB. See Can an 8 GB GPU Turn One Photo Into a 3D Model?.