Four Times the Context, Less Memory: How Six Models Store the KV Cache, Measured in llama.cpp and vLLM

Four Times the Context, Less Memory

We started Qwen3.5-9B in llama.cpp with a 128K context, and it allocated 4,146 MiB for the KV cache. Qwen3-8B, the previous generation from the same lab and the smaller of the two, had allocated 4,608 MiB for a 32K context. Four times the context, less memory, on the bigger model.

Parameter count does not explain that; the kinds of layers do. Qwen3-8B keeps every token in all 36 of its layers. Qwen3.5-9B keeps tokens in only 8 of its 32 layers, and the other 24 carry a small fixed state instead. Both numbers match a calculation from each model's config.json exactly. That is the subject of this post: six models whose KV caches are built in four different ways, how much each way costs as a conversation grows, and what to plan with for each. Every memory number here is either calculated from config.json or measured in llama.cpp and vLLM, and we say which.

If you want the attention designs themselves first (MQA, GQA, MLA, sparse and linear attention, with the models that use each), the previous post in this series covers them. The runtimes did not always agree with each other; where vLLM reserved more than llama.cpp, we give the number here and the reasons and settings in the next post.

Why caches differ: four kinds of layer

When a model reads a token, each attention layer keeps that token's key and value so later tokens can look back at it. What a layer keeps depends on its kind:

• Full attention keeps a key and a value for every token, for every KV head. Per layer, one token costs 2 (K and V) × KV heads × head size × 2 bytes. It grows with the conversation and never stops.
• Sliding-window attention only looks at the last W tokens, so it only needs to keep W of them. Its share should stop growing at W.
• Multi-head latent attention (MLA), used by DeepSeek, stores one compressed vector per token and rebuilds the heads' keys and values from it when needed.
• Linear attention (the Gated DeltaNet layers in Qwen3.5) keeps no tokens at all. It updates a fixed-size state as each token arrives.

Most recent models mix these. From config.json alone you can work out what one more token costs and what is fixed per conversation:

Two notes on the table. Gemma 3 4B's config lists only the window; its head counts and layer split are the values transformers fills in. The Qwen3.5 state counts the recurrent matrices and the short convolution buffers; llama.cpp keeps it in f32 and measured exactly 50.25 MiB. The models are not the same size: 4B to 20B total parameters, and two of them are mixtures of experts. This compares how they store context, not which model is better. Each design buys its memory saving with something else: a sliding layer cannot see past its window directly, and MLA spends compute rebuilding heads. We did not measure those trade-offs.

How we measured. llama.cpp build 4da6337, llama-server -ngl 99 -fa on --parallel 1 -c <length>, reading the KV cache sizes it prints at startup. vLLM 0.29.1rc1 with max_model_len set to each length and 85% of an A100 80GB, reading the cache memory and token capacity it prints. Lengths 4K, 32K and 128K, f16/bf16 cache. Most vLLM runs did not reach a ready engine: of 29 runs, 24 printed the cache lines and then failed at the sampling warm-up (a Ninja JIT build), and 2 were rejected at 128K because the length exceeds the model's limit (Qwen3-8B, Mistral). The KV cache is allocated before that warm-up, so we used the numbers as printed. Only the 3 DeepSeek runs reached a ready engine. We did not generate any text in vLLM, so nothing here says anything about its speed. Before running anything we committed the models, the calculation and eight predictions; the scorecard is at the end.

One thing to know about vLLM's numbers before reading them. vLLM reports a token capacity, and it computes it as "how many requests of the maximum length fit, times that length." So memory ÷ capacity is the memory vLLM reserves for one maximum-length request, divided by the length. For a plain model that equals the per-token cost. For models with fixed parts, the fixed part is spread over the length.

1. Full attention: the config is the whole story

Qwen3-8B: 36 layers × 2 × 8 heads × 128 × 2 bytes = 147,456 bytes, 144 KiB per token. At 32K that is 4,608 MiB.

Both runtimes matched the calculation exactly. llama.cpp allocated 576, 4,608 and 18,432 MiB at 4K, 32K and 128K; Mistral-7B-v0.3 got 512, 4,096 and 16,384. vLLM's per-token figure was within 0.01% of 147,456 and 131,072 bytes. 128K is past both models' maximum positions (40,960 and 32,768). vLLM refused to start; llama.cpp allocated the 128K cache but, by its own log, capped each conversation at the training length, so the extra memory bought no extra room. The next post has the details.

What to plan with. Only two settings change the memory: context length, and the cache's number format. Halve -c and you halve the cache. An 8-bit cache (-ctk q8_0 -ctv q8_0 in llama.cpp) took Qwen3-8B at 32K from 4,608 to 2,448 MiB, 53.1%. It is a little over half because each block of 32 values carries its own scale. We measured what it costs in quality in an earlier post (llama.cpp KV cache quantization). For these models, config.json is enough to plan.

2. Sliding window: the saving depends on the runtime

Gemma 3 4B: a layer costs 2 × 4 × 256 × 2 = 4,096 bytes per token. Five full layers grow (20 KiB per token). If the 29 sliding layers stop at 1,024 tokens, they hold 116 MiB, and the total at 32K is 756 MiB. If they kept everything, it would be 4,352 MiB. gpt-oss-20b's numbers are 771 and 1,536 MiB.

llama.cpp keeps the window plus one batch. At 32K it allocated 814 MiB for Gemma and 786 MiB for gpt-oss. With --swa-full, which makes the sliding layers keep everything, the figures were exactly 4,352 and 1,536 MiB. The default is the saving.

It is not exactly the window, though, and that surprised us. We had predicted gpt-oss would land above Gemma, as the calculation says (771 against 756). It landed below. The sliding layers hold the window plus one micro-batch (-ub, 512 by default), rounded up to a multiple of 256: 1,024 + 512 = 1,536 cells for Gemma, 128 + 512 = 640, rounded to 768, for gpt-oss. Gemma has 29 such layers, so its extra share is larger. We checked this after seeing the result by changing -ub: Gemma went 785 → 814 → 872 MiB and gpt-oss 780 → 786 → 798 MiB for -ub 256, 512 and 1024. Every value matched window + ubatch.

vLLM reserved much more. Started through the Python LLM class with max_num_batched_tokens at its A100 default of 8,192, for the same 32K request it reserved 2,682 MiB for Gemma and 1,155 MiB for gpt-oss. The sliding layers still stopped growing, but next to the window vLLM holds room for every token its scheduler can have in flight, 2 × max_num_batched_tokens, here 2 × 8,192 = 16,384 tokens. At these settings vLLM still saves 1,670 of the 3,596 MiB the sliding window could save, 46%. Lowering max_num_batched_tokens brings it back (Gemma at 512: 882 MiB per 32K request). That default depends on the GPU and on how vLLM is started (vllm serve on an A100 uses 2,048). The next post has the reserve formula, the measurements that check it and what the setting costs.

What to plan with. For a sliding-window model, plan with the runtime's reserve, not the window. In llama.cpp keep --swa-full off (it takes Gemma at 32K from 814 to 4,352 MiB). An 8-bit cache still halves what is left (Gemma 32K 432 MiB, gpt-oss 418 MiB in llama.cpp).

3. MLA: the saving holds in both runtimes

DeepSeek-V2-Lite: (512 + 64) × 2 bytes × 27 layers = 31,104 bytes, 30.4 KiB per token. If the runtime unpacked the latent into 16 heads with a 192-wide key and 128-wide value, it would cost 270 KiB, almost nine times as much. At 32K that is 972 against 8,640 MiB.

Both runtimes stored the latent. llama.cpp allocated 121.5, 972 and 3,888 MiB at 4K, 32K and 128K, exactly the calculation. vLLM measured 31,104, 31,102 and 31,105 bytes per token.

What to plan with. Nothing special for memory; an 8-bit cache halves it again (516 MiB at 32K in llama.cpp). The work was in getting there. We converted the model to GGUF ourselves with the converter from the same build, and it stopped because it did not recognize the tokenizer; we had to tell it which tokenizer it was. If you convert a DeepSeek model yourself, check that the cache size it prints matches the latent calculation. A file that unpacks the heads would still run; it would just cost nine times the memory.

4. Linear-attention hybrid: a fixed cost per conversation

Qwen3.5-9B: only 8 of 32 layers keep tokens, 2 × 4 × 256 × 2 × 8 = 32 KiB per token. The other 24 layers keep a fixed state per conversation.

llama.cpp showed both parts separately: KV of 128, 1,024 and 4,096 MiB at 4K, 32K and 128K (exactly 32 KiB per token), plus a fixed 50.25 MiB "recurrent" state at every length.

vLLM's per-token figure was 59,145, 36,444 and 33,668 bytes: 80.5%, 11.2% and 2.7% above 32 KiB. The excess shrinks with length, as a fixed cost spread over more tokens should. Worked back from vLLM's cache blocks, the fixed part is about 99 MiB per request, twice llama.cpp's 50 MiB: with prefix caching on, which is the default, vLLM gives each group of linear layers two blocks instead of one. With prefix caching off it fell to about 49.5 MiB (the next post shows the calculation and the run).

What to plan with.

• Context length matters less here than anywhere else in this post: 32K costs 1,074 MiB in llama.cpp, less than a quarter of Qwen3-8B's 4,608, and 128K costs 4,146.
• The fixed state multiplies with parallel conversations, not with length.
• An 8-bit cache shrinks only the 8 attention layers (1,024 → 544 MiB at 32K); the 50.25 MiB state stayed the same.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

Choosing a model and a length

Our predictions, scored

We committed eight predictions before measuring. Six were right.

P6 did not say at which length it would be judged. We chose 32K, this post's reference length, after seeing the results, and report all three. The -ub runs in section 2 were added after the main results to explain P7 and are not part of the scorecard; so were the vLLM runs described in the next post.

Limits

• One build of each runtime. Sliding-window and hybrid handling change between versions; the reserve formulas above are from these builds.
• Memory only. We did not measure speed or answer quality for any setting here, including the ones we recommend for memory.
• vLLM figures are the reserve for one maximum-length request, not the memory a real workload ends up using.
• Llama 3.1 8B was in our plan as the plain baseline; we could not get its config without a license agreement and used Qwen3-8B and Mistral-7B-v0.3.
• 128K is past the training length of Qwen3-8B and Mistral-7B-v0.3.