Pick llama-server's -c From Two Short Runs, Not a VRAM Estimate: Load Memory Within 1 MiB at Seven Checked Lengths, and the Estimate Overshot Gemma 3 at 128K by 13.7 GiB

Pick llama-server's -c From Two Short Runs, Not a VRAM Estimate

A common VRAM estimate, the one that assumes every layer is full attention, says Gemma 3 4B with a 128K context needs 20.9 GiB, which puts it on a 24 GB card. llama-server, started with that context, held 7.2 GiB. At 4K the same estimate was within 251 MiB of the measurement. Checked at a short length, the estimate looks right; the gap opens as the context grows.

So we stopped calculating and measured instead. Start llama-server once at 4K and once at 8K, and draw a straight line through the two memory readings. Read off the longest -c that fits your memory budget (we call it the max -c), and start once more at that length to check. At the seven lengths we started for the check, the line predicted load-time memory within 1 MiB. It also reproduced our earlier 32K and 128K readings to the MiB, though those were values we had already seen, not a blind test.

It missed once. Processing a prompt added 12 to 14 MiB, and that pushed two of the four lengths we first picked over budget, by 4 and 11 MiB. We now subtract 16 MiB more. This post is the corrected method, the numbers behind it, and where the estimate goes wrong.

This is part 1 of a series on fitting a local model to your GPU.

The setup

• Runtime: llama.cpp 4da6337, llama-server -ngl 99 -fa on --parallel 1 -c <length> -lv 4. Default f16 KV cache, one slot.
• Models: Qwen3-8B Q4_K_M (every one of 36 layers uses full attention), Qwen3.5-9B Q4_K_M (8 of 32 layers use full attention, the rest keep a fixed-size state), Gemma 3 4B it Q8_0 (34 layers, most of them sliding-window).
• Reading: the process's memory in nvidia-smi, three seconds after the server reported ready, plus the common_memory_breakdown_print line in the log. Every condition ran twice, and the two readings were identical every time.
• GPU: one A100 80GB. We did not run on 8, 12, 16 or 24 GB cards; instead we set memory budgets and compared the A100 readings to them. Step 3 below says how the budgets were set.

We committed our predictions before each measurement round: five about the estimate, four about the method. All nine are scored at the end.

The method: two runs, one line, one check

GPU memory is made of two kinds of pieces. Some are fixed whatever the context (model, about 480 MiB that no log line lists, Qwen3.5-9B's state, Gemma's window cache). The rest grow at a fixed rate per token (the full-attention KV cache). The compute buffer grows too, but in our runs it grew by 1 MiB per 1K tokens on every model (100, 128 and 224 MiB at 4K, 32K and 128K for Qwen3-8B), which is also a straight line. So two readings at two lengths are enough to draw the whole line. You do not need to know how many layers keep every token; the slope already contains it.

1. Start the server at -c 4096, wait for it to report ready, and read the process's memory: nvidia-smi --query-compute-apps=pid,used_memory --format=csv. Call it M1. Stop the server.
2. Do the same at -c 8192. Call it M2. The slope is (M2 − M1) ÷ 4,096, in MiB per token.
3. Set your memory budget L. Before starting the server, read the free memory with nvidia-smi --query-gpu=memory.free --format=csv and subtract 512 MiB, a reserve for the driver and anything else that starts later. We had no small cards, so we took the card size in decimal GB, converted it to MiB and subtracted 512: for 8 GB, 8 × 10⁹ ÷ 2²⁰ − 512 = 7,117 MiB. Real cards are sized in binary units, and an 8 GB card reports a total close to 8 GiB (8,192 MiB), so our budget is about 560 MiB more conservative than a real 8 GB card, and about 1.6 GiB more than a 24 GB one. "8 GB" in the tables below is the name of this budget.
4. Compute max -c = 4,096 + (L − 16 − M1) ÷ slope, round it down to a multiple of 1,024, and cap it at the limit in the GGUF metadata (n_ctx_train). The 16 MiB covers the 12 to 14 MiB we saw during prompt processing in this experiment; more on that below.
5. Start once more at that length and send a prompt as long as you will really use, watching the peak. If you will use long outputs, generate that much in the check run and measure it. We filled 90% of the context and generated only one token.

We picked 8K as the second point because 32K would not fit Qwen3-8B on an 8 GB card at all (9,668 MiB), and the method should work on the card you are sizing for.

Did the line hold?

The line through two points reproduced all six 32K and 128K readings exactly. We had predicted within 16 MiB. Those readings, though, came from the first round, so we had already seen them. A line through the first-round 4K and 32K readings had already hit 128K exactly, and that is where this method came from. The 32K and 128K readings were not used to draw the 4K/8K line, but this is not a prediction made blind. The blind test is the check runs in the next section.

Checking at the max -c

We computed the max -c for the 8, 12, 16 and 24 GB budgets from the line, before adding the 16 MiB, committed the predictions, and then started the server. Each distinct length ran twice with a prompt filling 90% of it; the peak is the largest nvidia-smi reading taken every 0.2 seconds.

The line predicted load-time memory within 1 MiB at all seven lengths. Processing the prompt added 12 to 14 MiB in all fourteen runs, and that was enough to push two cells over budget, by 4 and 11 MiB. Rounding the max -c down to a multiple of 1,024 had left those two with less room than the prompt needed.

We had predicted that every length picked this way would stay under budget, and it did not. The overshoot is inside the 512 MiB reserve, our budgets are more conservative than real cards as noted above, and we did not run on real cards, so this is not an observed out-of-memory error. But a method that picks the length should not depend on the reserve.

So we added the 16 MiB to step 4 and committed a prediction before running again: subtracting 16 MiB moves only those two cells down one step, and both should stay under budget. Both did:

This correction came after the miss, so it is a fix we tested once, not a prediction that held from the start, and the 16 MiB is what we saw under these conditions (A100, one prompt, one generated token).

The max -c for each budget

From the line, with the 16 MiB correction. Treat these as candidates rather than settings to copy: recompute with your card's free memory as in step 3, then check with step 5.

The last column uses a budget of 8,192 − 512 = 7,680 MiB on the same lines; we did not start the server at those lengths. At 12 GiB only Qwen3.5-9B changes, to 194,560, and everything at 16 GB and up is already at the cap.

The estimate column is calculated, not run: it is where the estimate's own line crosses the 8 GB budget. On our 8 GB budget the estimate picks contexts 3.8 and 5.0 times shorter than the line for Qwen3.5-9B and Gemma 3 4B. For Qwen3-8B it goes the other way and picks 16,384, where the line puts load-time memory at 7,348 MiB, 231 over our budget. That is inside the 512 MiB reserve and would fit an 8 GiB budget (7,680 MiB), so the large error is the too-short picks on the other two models.

Where the estimate goes wrong

The estimate we compared against is the simple arithmetic that assumes every layer stores K and V for every token:

total = GGUF file size + layers × KV heads × (K head size + V head size) × 2 bytes × context length

We did not test any particular online calculator; we put this arithmetic next to the measurements:

For Qwen3-8B the estimate is a few hundred MiB low. For the other two it is close at 4K and then overshoots more and more, until at 128K it asks for 2.2 times (Qwen3.5-9B) and 2.9 times (Gemma 3 4B) what the server used.

The log splits the Qwen3-8B 32K total into three pieces, and nvidia-smi adds a fourth:

| - CUDA0 (A100 80GB PCIe) | 81151 = 77113 + (9191 = 4455 + 4608 + 128) + -5152 |

• Model, 4,455 MiB, 340 MiB less than the 4,795 MiB file; 334 MiB of that sits in a CPU_Mapped model buffer. Qwen3.5-9B splits the same way. Gemma 3 4B does not: its GPU part is only 14 MiB under the file, and 680 MiB on the CPU comes on top. We did not check which tensors stay on the CPU.
• KV cache, 4,608 MiB, exactly the estimate's 144 KiB per token for this model.
• Compute buffer, 128 MiB. Across all nine conditions it was 79 to 224 MiB, the smallest of the four pieces.
• About 480 MiB in no log line. Process total minus the three logged pieces was 476 to 481 MiB in all nine conditions, constant for each model at every length. It looks like the CUDA context and libraries every CUDA process carries, but we did not measure that directly.

That explains the few hundred MiB on Qwen3-8B. The gap on the other two (12,097 and 14,009 MiB at 128K) comes from the assumption that every layer keeps every token:

• Qwen3.5-9B uses full attention in only 8 of 32 layers. At 128K the log reads 4096.00 MiB (131072 cells, 8 layers), plus a 50.25 MiB RS buffer that does not grow with context. The estimate counted 32 layers, 16,384 MiB.
• Gemma 3 4B keeps a full cache in 5 layers (2,560 MiB at 128K) and a 1,536-cell sliding-window cache in the other 29 (174 MiB at any length). We worked out why it is 1,536 cells (the 1,024-token window plus one 512-token micro-batch) in the llama.cpp vs vLLM memory post. The estimate counted 34 full layers, 17,408 MiB.

The slopes say it in one number each. The estimate's slope is 144 MiB per 1K tokens for Qwen3-8B, 128 for Qwen3.5-9B and 136 for Gemma 3 4B. The measured slopes are 145, 33 and 21; on all three models 1 MiB of that is the compute buffer and the rest (144, 32 and 20) is the KV cache. We measured these cache layouts for six models in the KV cache by model post.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

Our predictions, scored

Before the first round (the estimate):

Before the second round (the method):

P5 came from Gemma 3's 262K-word vocabulary: we assumed a bigger output layer needs a bigger compute buffer. It did not, and the logs did not tell us why. F4 is the miss that changed the method.

Limits

• One A100 80GB, Linux, one build. The 476 to 481 MiB and the 12 to 14 MiB may differ on another GPU or driver, and we did not run on a small card. Our budgets were set from decimal GB, not from a real card's memory.free. If your nvidia-smi does not show per-process memory, the log's breakdown line plus the unlisted part is a fallback, but we only know that part's size on this machine.
• f16 KV cache only. With an 8-bit cache the compute buffer grows much faster with context, as we found in the 8 GB context budget post, and we have not checked whether two points still give a straight line.
• Prompt processing, not long generation. Each check sent one prompt and generated one token, and memory was polled every 0.2 seconds; a peak shorter than that could have been missed.
• Three models. Other layouts (MLA, other window sizes) should still be lines if every piece is fixed or per-token, but we have not run them.
• The cap is a GGUF metadata value. "Cap" in the tables is the n_ctx_train written in each GGUF. For Qwen3-8B that is 40,960, while the model card describes a native context of 32,768 and 131,072 with YaRN. Read the Qwen3-8B 128K run as a memory number only.

Next

The line's starting point is mostly the model file. The next part changes that: Q8_0, Q4_K_M and Q3_K_L of the same model, how many tokens of context each gigabyte saved buys on these cards, and what it costs in accuracy. The newsletter below sends each part when it is out.