We Forced Qwen3.8 27B Into 8GB. It Failed the Same Task 9 Times.

We Forced Qwen3.8 27B Into 8GB. It Failed the Same Task 9 Times.#

Qwen3.8 27B's model page lists 27 billion parameters and an Apache 2.0 license, and aims it at coding and "long-horizon agentic tasks". We wanted all 64 of its layers on the GPU inside an 8 GB budget. It fits, at about 50 tokens a second on our A100.

Then we gave it the coding job we used to rank models for 8 GB. It averaged 0.462, below the much smaller Qwen3.5-9B at 0.781, which ran as a different file with a 16K f16 cache. On one of the four tasks it scored 0 in nine runs out of nine, across three settings, and never wrote to the file.

What we measured, and what we didn't#

Everything ran on an A100 80GB with llama.cpp build 1137. We read the server process's peak memory and compared it with a budget: 8,192 MiB, minus 512 for a desktop (our assumption), minus a 16 margin, which gives 7,664 MiB. Nothing was capped, so a real 8 GB card may behave differently. Every speed here is our machine's.

The command#

llama-server -m Qwen3.8-27B-UD-IQ2_XXS.gguf -ngl 99 -fa on \
  -ctk q4_0 -ctv q4_0 -c 10752 -ub 128 -t 8 --parallel 1 --jinja

The IQ4 file we downloaded is 14.3 GB on the model page (13.27 GiB), well past the budget. IQ2_XXS, the smallest 2-bit file in the unsloth repository, is 7.27 GB (6.77 GiB) and leaves about 735 MiB for everything else. The repository also has smaller 1-bit files, which we did not test.

Each row below is the same file. Note the prompt length column, because a longer prompt reads higher.

The first two rows change only -ngl. -ngl 57 puts 8 of the 64 layers on the CPU, because the output layer counts as one and goes to the GPU first; the load log says so. That fits the budget and takes decode from about 50 to 1.7 tokens a second, on 8 threads of an EPYC 7742; 32 threads reached 5.2. A desktop CPU will differ, and we did not measure one. For the same reason -ngl 64 is not everything on this file: it leaves one layer on the CPU, and that row read about 12 tokens a second with a shorter prompt, still over budget.

The rows after that change more than one flag, so don't read any step as the effect of one flag. The 11,264 row looks fine until real work: its peak in agent runs was 7,666 MiB, 2 over. At 10,752 the peak in agent runs was 7,654, 10 to spare.

What those cuts give up:

• A coarser cache. On a different model, Qwen3-8B, q4_0 raised perplexity by 0.5% to 1.1% against f16 in our KV cache quantization guide. We did not measure that on this file.
• Slower prompt reading. From about 1,250 tokens a second in row one to about 1,010 in the last row, with cache, context and batch all changed together.
• Context. About 10K tokens is little room for a coding agent.

The job#

These are the same four small Python tasks from our 8 GB test: add due dates to a todo app, fix a bug with a misleading traceback, add stock reservations across two files, and write a version parser from a spec. Each ran three times, with the same small agent, a limit of 30 turns and 30 minutes. The score is the share of hidden tests that pass, tests the model never sees. A score of 1 means they all pass.

The tasks, the agent and the limits match across rows. The 9B row is a different model and file with a different cache and context, so this compares setups, not one flag or one generation.

The top two rows differ by 512 tokens of context and 0.177 in the mean. All of that gap is T3, and with three runs per task we can't tell run-to-run spread from a context effect. Of the 27B rows, only the top one stays inside the budget in real runs.

The time gap has two parts. 27B's median decode at our setting was 48.3 tokens a second against 125.2 for 9B, and it generated about three times as many tokens (486,417 against 159,743). About 75 of its 183 minutes were T4. The scored runs left threads at the default, 64 on this machine, rather than the -t 8 in the command above; with every layer on the GPU we expect little difference, but we did not compare them.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

One task, nine runs#

T4 asks for four functions from a README: parse, compare, satisfies and max_satisfying. It had the lowest mean of the four tasks in our earlier test. This is the start of the first run at the setting that fits, shortened to the file names:

[1] read_file README.md
[1] read_file semver.py
[1] read_file test_semver.py
[2] context 10,752 / 10,752
[2] over half the context: cleared 4 old messages
[2] the answer was cut off (length)
[3] context 1,210 / 10,752
[3] read_file README.md
[3] read_file semver.py
[3] read_file test_semver.py
[4] context 10,752 / 10,752
[4] over half the context: cleared 4 old messages
[4] the answer was cut off (length)

Across the three 27B settings, that's nine runs. None sent a write or an edit, each hit the cut-off between 5 and 13 times, and each scored 0 of 10. Eight ended at the 30-minute limit and one at the 30-turn limit. On the other three tasks, at the same settings, a file was written or edited in 24 of 27 runs.

On the same task, in its first run, 9B read the same three files, wrote all of semver.py on its next turn, and ran the tests. Across three runs it sent 8, 11 and 9 write or edit calls and scored 0.7, 0.4 and 0.4.

In the two settings with more room, nearly all of the distance to 9B is this task: T4 accounts for 0.125 of the mean, and the gaps are 0.142 and 0.114. At the setting that fits, the gap is 0.319, and two bad T3 runs add to it.

More room didn't change T4#

A 16K f16 cache, which we ran only to check because it is 924 MiB over budget, didn't change the T4 score, and that context filled too: in all nine runs, every cut-off came with a full context. We tested no cause separately, and we did not score the IQ4 file or a context above 16K.

Qwen3.5-27B scored 0.953 on these tasks, but it is another generation at Q4_K_M with a 17,116 MiB peak, so it says nothing about this file.

Limits#

Four small Python tasks, three runs per task, one small agent, with memory read on an A100 against a budget. Other processes shared the A100 during one T4 run in each of the two over-budget rows. The fitting setting had the GPU to itself. We did not test chat, other languages or larger codebases.

What to do with this#

If a large model loads on your card, run your own task at that exact setting, next to a smaller model, before you keep it. For a smaller model to start from, and how nine models scored against 8, 12, 16 and 24 GB budgets, see our 8 GB test.