Claude Code on a Local Qwen3.8-Flash-Next on a 12 GB GPU Budget: llama.cpp or Strata

Before Claude Code reads a single file of your project, it sends the model about 28,000 tokens: its system prompt and the descriptions of its tools. We pointed it at Qwen3.8-Flash-Next running locally on a 12 GB budget. With llama.cpp, that first request took 139-161 seconds before the model wrote anything. With Strata, the same request took about 12 seconds. The gap did not close after that: llama.cpp wrote its answers at about 7 tokens per second, Strata at about 35 (with its speculative decoding on, the default; we worked these speeds out after the runs, from each engine's own log lines).

This guide walks through both setups, step by step, based on the commands we ran (differences from our test harness are in the setup note at the end), and shows what each one costs you in startup time, waiting and memory. Pick one route; you do not need both. If you pick Strata, add one more variable: we first ran without it, and both small-feature runs ended in an error (section 6).

0. What you get

Claude Code is Anthropic's coding agent for the terminal. It talks to its model through one HTTP endpoint, /v1/messages. Both llama.cpp's llama-server and Strata serve that endpoint themselves, so Claude Code can talk to them directly, with no proxy such as LiteLLM in between. You set a few environment variables and start claude as usual.

For background on llama-server and Strata themselves, see our Strata vs llama.cpp post.

Claude Code in the terminal sends each request to a local server at 127.0.0.1:8080 instead of Anthropic's API; llama-server or Strata answers on /v1/messages, with part of the model on the GPU and the experts in system RAM.

The two routes, measured on the same machine with the same model file and task:

	① llama.cpp	② Strata
Install	build one binary	build a Docker image
Server start	9-20 s	about 30-50 s in 6 of the 11 starts that came up, 4-10 min in the other 5 (cause not identified)
First request (about 28K tokens)	139-161 s	about 12 s
Small bug fix, whole task	280-370 s (3 runs)	46-58 s (3 runs)
Small feature, hidden tests passed	6/6 in both runs, 1,136 s and 1,417 s	6/6 in both runs with the output cap, 184 s and 190 s
Peak GPU memory	10,994 MiB	11,670 MiB

One bar per run, twelve runs: bar length is the whole task, the dark part is reading Claude Code's first request of about 28K tokens. On the small bug fix, llama.cpp took 280, 289 and 370 seconds, of which 139-161 seconds went to the first request; Strata took 46-58 seconds, about 12 of them on the first request. On the small feature, llama.cpp took 1,136 and 1,417 seconds; Strata's two runs without an output cap stopped on an API error at 137 and 149 seconds (hatched), and the two with an 8192 cap finished in 184 and 190.

One thing to know before the numbers: we do not own a 12 GB card. We ran on an A100 80GB and held each engine to about 12 GB of it, which is how the earlier Strata vs llama.cpp post measured too. Your times will differ: the GPU is faster than a gaming card, while the 8 server cores (2.25 GHz) may be slower than a desktop CPU, so read this post for what works within 12 GB rather than for exact times. Strata's own README says its first message takes "about 1 minute per 30,000 tokens"; we saw about 12 s. The CPU side was 8 cores of an AMD EPYC 7742, to match a desktop. The machine is shared with other work (1-minute load median 23-49 across our runs): the slowest llama.cpp run (370 s) and one Strata run overlapped with another program on the same GPU (9,920 MiB), and one Strata small-feature run shared it with a 414 MiB process.

1. What you need (both routes)

An NVIDIA GPU with 12 GB. Measured per engine process, llama.cpp peaked at 10,994 MiB and Strata at 11,670 MiB. 12 GB is 12,288 MiB, so Strata leaves about 600 MiB by its own processes, or 535 MiB by the whole-GPU peak (11,753 MiB). A real 12 GB card also loses memory to the display and driver, and we did not test on one.
64 GB of system RAM. llama.cpp's process was at 47.0-50.4 GiB at the end of each run (most of it the memory-mapped model file); Strata's container used about 37-40 GiB. We did not test machines with less.
Disk: about 70 GB for llama.cpp, about 90 GB for Strata (its own copy of the model, its setup data and the Docker image), so about 160 GB if you try both. The model: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, folder IQ2_XS, two files.
Claude Code (we used 2.1.302).

Why this model fits at all: Qwen3.8-Flash-Next has 125B parameters, but each token uses only 10 of the 512 experts in each of its 48 layers. The experts can stay in system RAM and the GPU holds the parts every token needs. Think of a person taking an exam: everything they learned is in their head, but each question only pulls out the part it needs. Both routes rely on this.

Everything a person has learned drawn as a grid of 512 cells, one exam question as a card, and the knowledge that question pulls out as 10 cells. Handwritten labels turn one at a time into the real terms: everything learned is the 512 experts in each layer, the question is one token, the knowledge pulled out is the 10 experts used. The line at the bottom says each token works with 10 of the 512, so the other 502 can wait in system RAM.

Download the model for route ① (route ② downloads its own copy into a Docker volume, see 3-2):

pip install -U huggingface_hub
hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF --include "IQ2_XS/*" --local-dir ~/models/qwen38

2. Route ① llama.cpp

Good: one binary, started in 9-20 s in all of our runs, easy to stop and start again, the same tool you may already use for other models.

Bad: with most experts on the CPU, it reads the first 28K-token request at about 200 tokens per second on 8 cores, so the first Claude Code session after starting the server waits 139-161 seconds before any work begins. (llama-server can reuse the matching start of the prompt in later sessions on the same running server; we did not measure that case.)

2-1. Build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout d89651a          # the commit we measured
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --target llama-server

2-2. Start the server

./build/bin/llama-server \
  -m ~/models/qwen38/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf \
  -c 65536 -np 1 -fa on -ngl 99 --n-cpu-moe 42 --port 8080

What each part does:

-ngl 99 --n-cpu-moe 42: put every layer on the GPU, then move the experts of 42 of the 48 layers back to the CPU. This is the knob that sets your GPU memory. A smaller number keeps more experts on the GPU, which is usually faster and uses more memory. 42 peaked at 10,994 MiB for us.
-c 65536 -np 1: a 64K context and one slot. With more slots, llama.cpp splits the context between them, and Claude Code's 28K-token start would not leave much room.
-fa on: flash attention, which keeps the attention buffers small.

The model's 48 layers drawn in a row: with -ngl 99 every layer's attention and shared weights sit on the GPU; --n-cpu-moe 42 moves the expert weights of layers 0 to 41 to system RAM, and only the last 6 layers keep their experts on the GPU.

When it prints listening on http://127.0.0.1:8080, go to section 4.

3. Route ② Strata

Strata is an inference engine built for this one model family. It decides by itself which experts to keep on the GPU and moves them as it runs.

Good: the first request took about 12 seconds instead of 139-161, and on average whole tasks finished 5.9 times faster for the bug fix and 6.8 times faster for the small feature in our runs (the feature runs with the output cap).

Bad: it needs Docker and an image build; its start time swung widely on our machine: of the 11 starts that came up, 6 took about 30-50 seconds and the other 5 took 4-10 minutes, and we did not identify why (llama.cpp took 9-20 s every time; the Strata times exclude a 10 s wait in our script); and it takes all the GPU memory left when it starts, which matters if anything else uses the card.

3-1. Build the image

You need Docker, the NVIDIA Container Toolkit and driver 580 or newer.

git clone https://github.com/Niko1221/Strata
cd Strata
git checkout 6f32ec0          # the commit we measured
docker build -t strata .

3-2. Start the server

docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v strata-data:/data -e MODEL=IQ2_XS -e FAMILY=qwen -e CONTEXT=65536 strata

The first start downloads the model (about 70 GB) into the strata-data volume and runs Strata's setup, which picks settings for your card. Later starts skip both. Keep the volume: Strata looks for its settings there, and without them it runs setup and downloads again (we hit this with a hand-made config).

Without a VISION setting, as in the command above, Strata installs with image input off (no), which is what you want: Claude Code does not send images. Our volume, though, was set up for the earlier post with images on, so the Strata numbers here were measured with image input on. We did not measure GPU memory with it off.

The server is ready when the log prints ready: http://127.0.0.1:8080/v1 (… Anthropic: /v1/messages …).

4. Connect Claude Code (both routes)

Open a new terminal and set six variables (seven for Strata) before starting claude:

Two terminals. In the top one llama-server starts; when Claude Code in the bottom one is given a bug fix, the top terminal counts through the first request from 2,048 to 28,380 tokens. Reading the 28,384-token first request took 141 seconds; the next request, 260 tokens, took 3.2 seconds. The bottom terminal prints 3 tests passed, 7 turns, 288.9 seconds. Selected lines from our run's logs, with the timestamps and prefixes cut and the port shown as 8080. The `$` command lines were written for illustration, not taken from the logs.

The screen above shows selected lines from the real logs of one of our llama.cpp runs on the bug-fix task. The run itself used port 18093, and in non-interactive mode Claude Code prints only its result.

export ANTHROPIC_BASE_URL=http://127.0.0.1:8080
export ANTHROPIC_AUTH_TOKEN=local
export ANTHROPIC_MODEL=claude-sonnet-4-5
export ANTHROPIC_SMALL_FAST_MODEL=claude-haiku-4-5
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export API_TIMEOUT_MS=1800000
cd ~/your-project
claude

On route ② Strata, also set this one. The reason is just below and in section 7.

export CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192

ANTHROPIC_BASE_URL sends every request to your server instead of Anthropic.
ANTHROPIC_AUTH_TOKEN can be any text. Neither server checks it unless you set a key yourself. With it set, Claude Code did not ask us to log in (checked in non-interactive mode).
ANTHROPIC_MODEL and ANTHROPIC_SMALL_FAST_MODEL are names only. Both servers answer with the one model they loaded, whatever name arrives. Claude Code uses a second, smaller model for some background work; we set that name too, to be safe.
CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 turns off telemetry, error reports and update checks.
API_TIMEOUT_MS=1800000 gives each request 30 minutes. The first request on route ① takes minutes; we set this so that a slow request is never cut off.
CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192 (route ②): Claude Code asks for up to 32,000 output tokens on every request. Strata refuses, rather than trims, any request whose conversation plus that limit is over the context (64K), so once the conversation passes about 33,500 tokens every request fails. A cap of 8192 lets the conversation grow to about 57,000 tokens. The llama.cpp runs were measured without this variable and had no such error.

We ran Claude Code in its non-interactive mode (claude -p "…") so every run got the same prompt. The same variables should work for the interactive screen, but we only measured non-interactive runs.

If Claude Code's interactive screen is new to you, our free lecture Your first session: ask, act, review walks through one session from the first question to reviewing the edit. It is about 7 minutes and needs no sign-in.

SOTAAZ course

LLM Quantization and Compression Hands-On — This course alone $79 · 24 lectures · first 3 free

This course alone $79 · 24 lectures · first 3 freeWatch free →

5. Check that the local model is really answering

Three things to look at during your first request:

The server log is busy. llama.cpp prints lines like prompt processing, n_tokens = 2048, progress = 0.07 and counts up to about 28,000. Strata prints reading the prompt: 6,912 of 28,155 tokens, 3 s so far. If the log stays quiet while Claude Code thinks, the request went somewhere else.
The GPU is in use. nvidia-smi shows llama-server or Strata's process holding about 11 GB.
Later turns are fast. After the first request, the server keeps what it read and only reads the new part. In one llama.cpp run the requests after the first were 65-813 tokens each and took 1.7-5.4 seconds.

Claude Code's summary will still show a price (total_cost_usd was about $0.14-0.16 per toy run). Nothing was billed. Claude Code multiplies the token counts by the list price of the model name in ANTHROPIC_MODEL.

6. A real job: a small feature

A two-bug toy task says little, so we also asked for a feature in a 93-line todo CLI (todo.py, 3 tests). The prompt asked for six things:

add takes --due YYYY-MM-DD and stores it on the item.
list prints (due YYYY-MM-DD) after the title.
A new list --overdue shows only items that are not done and are past their due date.
Today's date can be passed in, so it can be tested.
An invalid date is an argparse error (exit code 2).
Todo files from before the change, with no due dates, still load.

We graded with 6 tests the model never saw: storage, output format, the overdue filter, invalid dates, old files, and the old behaviour. The original code passes 2 of the 6; our own reference solution passes all of them.

Run	Turns	Time	Ended	Hidden tests
llama.cpp 1	16	1,136 s	clean	6/6
llama.cpp 2	20	1,417 s	clean	6/6
Strata 1, no cap	9	137 s	API error	6/6
Strata 2, no cap	9	149 s	API error	6/6
Strata 3, cap 8192	14	184 s	clean	6/6
Strata 4, cap 8192	14	190 s	clean	6/6

Why Strata's first two runs failed

Strata's first two runs stopped at turn 9 when Claude Code got this back:

API Error: 400 prompt (34835 tokens) + max tokens (32000) exceeds the context (65536); requests are never truncated.

The cause is the output cap Claude Code attaches to every request. It always asks for room to write up to 32,000 tokens (max_tokens: 32000); Strata's log shows of max 32000 on every answer. Strata adds that cap to the conversation so far, compares the sum with the 64K context (65,536 tokens), and rejects the request outright if it does not fit. It never trims the request. So once the conversation passes 65,536 minus 32,000, about 33,500 tokens, no request gets through, however short the actual answer would have been.

Both runs died right at that line. In the first, the last request whose size the log recorded was 30,104 tokens; five more short requests were served after it, and then a 34,835-token request was refused. The second was served at 33,108 tokens and refused at 35,037. These were non-interactive runs (claude -p), so they simply ended. In an interactive session the conversation only grows, so every message after that point would have hit the same error. The code had already been fixed, so the hidden tests passed, but the session itself was dead.

llama.cpp received the same cap from the same Claude Code settings and did not refuse it. Its two conversations grew to 37,391 and 40,465 tokens and finished without an error. That is why someone who only tried llama.cpp would never run into this.

The fix is to lower the cap Claude Code asks for. With CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192 the cap is 8,192 tokens, so the conversation can grow to 65,536 minus 8,192, about 57,300 tokens. The exact line is 8 tokens lower: in the first error Strata said to send a cap of at most 30,693, and 34,835 plus 30,693 is 65,528. Before rerunning we committed predictions (both runs end cleanly, 6/6, inside 100-300 s, Strata GPU memory at or under 11,800 MiB), and all four held. Those conversations reached 35,946 and 35,970 tokens, big enough to have failed without the cap.

The lower cap cost nothing on this task: the longest answer was 1,899 tokens, far under 8,192. We did not test a job that has to write more than 8,192 tokens in one go. Strata's error message also points to another fix, adding "fit_max_tokens": true to the model's strata-<model>.json, which it says shrinks the cap to whatever room is left. We did not test that either.

All four clean runs passed every hidden test, and the model wrote 4-6 tests of its own and kept the visible suite green. Time: 19-24 minutes on llama.cpp, a little over 3 minutes on Strata. The llama.cpp runs also took more turns; we did not analyse why. Two clean runs per engine only tell you that this size of job gets done.

7. Traps we hit while measuring

When the commands are right and it still does not work, check this table. We hit four of these five ourselves while measuring; the tool-call row is a symptom we did not see, but a different model can cause it. None of them is obvious from the error alone. The first one is the nastiest: it runs fine for a few turns and then breaks, so it looks like the model's fault rather than a setting.

Symptom	Fix
With Strata, a few turns go fine, then `API Error: 400 prompt (… tokens) + max tokens (32000) exceeds the context (65536)` appears and every request after it fails	Set `export CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192` before starting Claude Code. We reran twice with it and both runs ended cleanly. Strata's error also suggests adding `"fit_max_tokens": true` to the model's `strata-<model>.json`; we did not test that
The first request is a few hundred tokens longer than expected, and the agent follows rules you did not give it	Claude Code also reads `CLAUDE.md` files in the folders above your project. Work in a folder that is not inside another repository, or check what `CLAUDE.md` files sit above it. In our runs this took the first request from 28,383 to 29,306 tokens
Strata stops at start with "no VRAM is left for the expert cache"	Strata sizes its expert cache from the GPU memory that is free when it starts, so close other GPU programs first. If that is not enough, reinstall with smaller settings. On a volume that is already set up, settings only change with `-e REINSTALL=1` (the downloaded model is kept). The options the engine's error message lists that the Docker image exposes: `-e KV=q4_0` (a 4-bit KV cache), `-e VISION=cpu` (the image encoder on the CPU) or `no`. It also suggests a smaller context (`-e CONTEXT=…`), but Claude Code needs at least about 36,500 tokens of it: the first request is about 28,300 tokens and the output cap at least 8,192, so a 32K context fails on the first request. We did not test the other two at 12 GB
The connection check passes, but when you give it work it describes the fix in prose instead of editing the file, or prints text that looks like a tool call	Claude Code reads files, edits them and runs commands entirely through tool calls. A reply to a short greeting proves the connection, not that the model can work. The tool list arrives as text in the request, so no model is retrained per harness. The model does need the habit of writing a call in a fixed wrapper (such as `<tool_call>{...}</tool_call>`) and then waiting for the result, and the server has to recognise that wrapper. Most recent instruct models pick this habit up during tuning; base models and older models often break the format. In the llama.cpp build used here the server applies the chat template stored in the model file by default (`--jinja` is on), and tool calls go through that template. Qwen3.8-Flash-Next made tool calls fine on both engines, so we never saw this symptom. We did not test other models; if you switch, follow the section 5 check with a small task that has to edit one file
Times jump around between runs	With experts on the CPU, both engines slow down when other work uses the cores. Check the load with `uptime` or `htop` before you measure. Our machine is shared, and the median 1-minute load average per run was 23-49

8. Which one to pick

llama.cpp finished all 5 of its jobs cleanly. Strata finished the 3 bug fixes as is, and the small feature once the output cap was set. What decides it is the waiting and the spare memory.

If you keep Claude Code open and give it many jobs, take ② Strata with CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192. At its fastest it starts about 10-20 seconds slower than llama.cpp (5 of the 11 Strata starts that came up took 4-10 minutes, for reasons we did not identify), but wins back more than 2 minutes on the first request alone, and whole tasks were 5.9-6.8 times faster. The longest conversation we measured was about 36,000 tokens; much longer sessions are untested. We also did not test a real 12 GB card: on the 80 GB card Strata peaked at 11,753 MiB (535 MiB to spare), and because it sizes its cache from the free memory at start, a real card may give it less cache and a different speed.
If you cannot use Docker, or worry about memory headroom on the card, take ① llama.cpp. It leaves about 1,300 of 12,288 MiB free and got every job done, slowly. The question is whether you can wait around 20 minutes for a small feature.

If you read one more post, make it picking -c from two short runs, which shows how much context your own card can hold.

One-screen summary (route ①)

hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF --include "IQ2_XS/*" --local-dir ~/models/qwen38
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout d89651a
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server
./build/bin/llama-server -m ~/models/qwen38/IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf \
  -c 65536 -np 1 -fa on -ngl 99 --n-cpu-moe 42 --port 8080
# new terminal
export ANTHROPIC_BASE_URL=http://127.0.0.1:8080 ANTHROPIC_AUTH_TOKEN=local \
  ANTHROPIC_MODEL=claude-sonnet-4-5 ANTHROPIC_SMALL_FAST_MODEL=claude-haiku-4-5 \
  CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 API_TIMEOUT_MS=1800000
cd ~/your-project && claude

Setup: one A100 80GB (driver 580.178.04), each engine held to about 12 GB (llama.cpp --n-cpu-moe 42, Strata --vram-reserve-mib 69800, which leaves that much of the 80 GB unused), AMD EPYC 7742 limited to 8 cores (taskset -c 0-7 … -t 8 / --cpuset-cpus 0-7), 251 GB RAM, model files on a rotational disk, machine shared with other work. Claude Code 2.1.302 with a fresh home folder, non-interactive mode, tools limited to Read, Edit, Write, Glob, Grep and python/pytest in Bash. llama.cpp d89651a; Strata 6f32ec0 Docker image. Seven predictions were committed before the runs; six held (P5 and P6 were judged including the two Strata runs that ended in an API error; P6 also holds on the clean rerun; P4's memory limits judged by the engine processes; by whole-GPU memory one llama.cpp run reached 11,003 MiB, just over its 11,000 limit). One Strata bug-fix run (46.2 s) was faster than the predicted range (50-120 s). The output-cap rerun had four predictions of its own, committed before it ran; all held. Our harness differed from the commands above in these ways: Claude Code ran under env -i with DISABLE_TELEMETRY=1, --permission-mode acceptEdits and a timeout; llama.cpp also had -t 8 --no-webui and its own port; Strata reused the earlier post's volume and models folder (-v …:/models), one GPU (--gpus device=0), image input on, and the 64K context already in its config; it also ran on a different port, as a detached named container (-d --name), with its config file copied from the default and --vram-reserve-mib set before every run. Pilot runs 2026-10-23 (the Strata log line in section 5 and the VRAM start failure in section 7 come from them), measurements 2026-10-24 (the terminal GIF, the llama.cpp log line in section 5 and the 400 error in sections 6 and 7 come from these).