DeepSeek's Config Says 16 KV Heads, Its Cache Holds a Ninth of That: Seven Ways LLMs Handle Attention, and Which Models Use Each

DeepSeek's Config Says 16 KV Heads, Its Cache Holds a Ninth of That

DeepSeek-V2-Lite's config.json says the model has 16 key-value heads in each of its 27 layers. The usual way to size a KV cache assumes every token stores a key and a value for every KV head in every layer, in 16 bits (2 bytes). When keys and values are the same size, that is 2 × KV heads × head size × 2 bytes × layers × tokens (the first section below explains where it comes from). In DeepSeek-V2-Lite each head's key is 192 wide (including 64 for position information) and its value 128, so the count is (192 + 128) × 16 × 2 bytes × 27 layers × 32,768 tokens: 8,640 MiB for one 32K-token conversation. Then we started the model in llama.cpp, and it allocated 972 MiB. vLLM reserved the same amount: 31,102 bytes per token, which over 32,768 tokens is also 972 MiB. Both runtimes used a ninth of the calculation. We had not misread the heads. The model simply never stores them. It stores one compressed 576-number vector per token per layer and rebuilds the 16 heads from it on the fly. 576 numbers × 2 bytes × 27 layers is 31,104 bytes per token, within 0.01% of what vLLM reserved.

That one gap is a good way into a larger fact: the attention layers of 2026's open models are not one design. Some store every head, some share heads, some forget old tokens, some compress each token, and some keep no tokens at all. The choice decides how much memory a long conversation costs, how fast the model decodes, what it can recall from far back, and which runtimes and tricks work with it.

This post walks through the seven designs you will meet in current models: multi-head, multi-query, grouped-query, sliding-window (and its local-global mixes), multi-head latent, sparse, and linear-attention hybrids. For each one we cover what it stores, which models use it, and what it gives up. Every model-to-design mapping below was checked against that model's config.json. Three Meta models (Llama 3.1, Llama 3.3 and Llama 4 Scout) are gated behind a license agreement, so for those we read the identical config in Unsloth's public copy. Some configs leave fields out, Gemma 3 4B's among them; there we took the values Hugging Face transformers fills in, and say so where it matters; the memory numbers are calculated from those files for a 32K-token conversation in 16-bit precision. A later post in this series, measured in llama.cpp and vLLM, checks six of these calculations against what two runtimes actually allocate.

What attention keeps, and the two bills it sends

When a transformer generates token number 10,001, each attention layer compares that token's query with the keys of all 10,000 earlier tokens and takes a weighted mix of their values. Recomputing those keys and values every step would be wasteful, so the runtime keeps them: that is the KV cache. (Our KV cache primer covers the basics with code.)

The cache sends two bills:

• Memory. Every token adds, per layer, a key and a value for each KV head. For a plain layer that is 2 × KV heads × head size × 2 bytes. Multiply by layers and by conversation length.
• Bandwidth and compute. Each new token has to read the whole cache once per layer. At long context, decoding is limited by how fast the GPU can pull those bytes out of memory, not by arithmetic.

MHA, the first design below, is the baseline. The six after it each cut one of five things:

1. how many KV heads are stored per token (MQA, GQA),
2. how many tokens are kept (sliding window, chunked attention),
3. how big each token's entry is (MLA),
4. how many stored tokens are read per step (sparse attention),
5. whether tokens are stored at all (linear attention).

The figure shows the four of those five that change memory; sparse attention (the fourth item) keeps the whole cache, so it has no bar. It puts one model from each family on the same axis. The models are not the same size (4B to 48B total parameters), so read it as "how this design stores context," not "which model is better."

1. Multi-head attention (MHA): every head keeps its own keys and values

This is the original transformer layer. Each of the H heads has its own query, key and value projection, and each one keeps its own keys and values for every token.

Who uses it. Mostly older or smaller models now. Llama 2 7B (32 query heads, 32 KV heads) and Gemma 7B (16 and 16) are MHA. Among the models we checked, no 2025-2026 release uses plain MHA in its attention layers.

What it costs. Llama 2 7B stores 2 × 32 heads × 128 dims × 2 bytes × 32 layers = 512 KiB per token. A 32K conversation would need 16 GiB of cache, more than the model's 12.6 GiB of 16-bit weights (Llama 2 was trained to 4K, so this is a thought experiment, but it shows why nobody ships long-context MHA at this size).

What it buys. Every head can attend with its own keys, which is the most expressive option, and every runtime, kernel and optimization supports it. It is the baseline every other design is measured against.

2. Multi-query attention (MQA): all heads share one key and one value

Noam Shazeer's 2019 proposal: keep many query heads, but give all of them one shared key head and one shared value head. The cache shrinks by the number of heads.

Who uses it. Falcon 7B (71 query heads, multi_query: true, head size 64), Google's PaLM, Gemma 3 1B (4 query heads, 1 KV head), and the attention layers of AI21's Jamba Reasoning 3B (20 query heads, 1 KV head).

What it costs. Falcon 7B stores 8 KiB per token, 256 MiB for 32K. That is a 64th of Llama 2 7B's.

What it gives up. One key per token has to serve every head, so the model loses some of its ability to look for different things at once. Shazeer reported only minor quality loss, and PaLM's authors found it neutral at their scale, but the GQA paper reports quality degradation and training instability: MQA models trained from scratch had frequent loss spikes. There is also a practical cost at scale: when a model is split across GPUs by heads (tensor parallelism), a single KV head cannot be split, so every GPU holds a full copy of it.

3. Grouped-query attention (GQA): the compromise that became the default

Ainslie et al. (2023) proposed the middle ground: split the query heads into G groups, and give each group one shared key head and one value head. G = 1 is MQA, G = H is MHA. Their paper also showed that an existing MHA checkpoint can be converted to GQA with a short extra training run, about 5% of the original pre-training compute.

Who uses it. Almost every dense model released since 2024, and the full-attention layers inside most of the hybrids below:

What it costs. Llama 3.1 8B stores 128 KiB per token, a quarter of Llama 2 7B's, so a 32K conversation needs 4 GiB. That is still the largest bar among the modern models in the figure, and it is why everything below exists.

What it gives up. Less than MQA. The GQA paper reports uptrained GQA reaching quality close to MHA at speed comparable to MQA. The group count is a dial, and the table shows how differently teams set it: Gemma 3 27B keeps 2 queries per KV head, Qwen3-235B shares one KV head across 16. GQA does nothing about the other axis, though: the cache still grows with every token, in every layer, forever.

4. Sliding window and local-global mixes: forget old tokens in most layers

A sliding-window layer only attends to the last W tokens. It never looks further back, so it only needs to keep W tokens. Its share of the cache stops growing once the conversation is longer than W.

The catch is that a layer with a window cannot directly see anything older than W. Information can still travel further by hopping: layer 2 sees what layer 1 summarized from its window, and so on. But that relay is lossy, so the current designs keep some layers that see everything.

Who uses it, and how. The mixes differ a lot, and the config fields that say so differ too:

• Mistral 7B v0.1: a 4,096-token window on all 32 layers (sliding_window: 4096). Mistral 7B v0.3's config sets sliding_window: null, so that version is plain full attention. The config does not say why it was dropped.
• Gemma 2: alternates one sliding layer (window 4,096) and one global layer that attends to the whole sequence, every other layer (21 and 21 in the 9B).
• Gemma 3: five sliding layers for every global one, with a much smaller window: 1,024 tokens in the 4B and 27B, 512 in the 1B. Only the 1B's config spells the pattern out (sliding_window_pattern: 6). The 4B's config lists neither the pattern nor its head counts, so those come from transformers' defaults: 29 sliding and 5 global layers, 4 KV heads of 256 dimensions. llama.cpp's load log for the 4B shows the same numbers. The 1B is also MQA (one KV head), so it uses two of the designs at once.
• gpt-oss-20b and 120b: strict alternation, half sliding and half full (layer_types lists 12 and 12 in the 20b, 18 and 18 in the 120b), with a window of only 128 tokens.
• Llama 4 Scout: a variant called chunked attention. Three of every four layers attend only within fixed 8,192-token chunks (attention_chunk_size: 8192), and every fourth layer is global and uses no positional encoding at all (12 of 48 layers). Watch the field name: in no_rope_layers, a 1 means the layer uses RoPE; the 12 zeros are the global layers. Meta's announcement names the design iRoPE; the chunk size and the one-in-four pattern come from the config and Hugging Face's release notes.

What it costs. The global layers still grow per token; the sliding layers are capped. For Gemma 3 4B, 5 global layers cost 20 KiB per token and the 29 sliding layers cost a flat 116 MiB once the window is full: 756 MiB at 32K, against 4,352 MiB if every layer kept everything. gpt-oss-20b comes to 771 MiB against 1,536. Mistral 7B v0.1 stops growing at 512 MiB, whatever the length.

What it gives up.

• Reach in most layers. Only the global layers can look up a token from 30,000 tokens ago directly. Gemma 3's technical report found minimal perplexity change across local-to-global ratios up to 7:1 and when shrinking the window to 1,024; that is the authors' measurement on their own evaluations. Perplexity is an average, and it says little about whether one specific fact from far back can still be retrieved.
• The saving depends on the runtime. The architecture permits the saving; the runtime has to take it. In a later post on deployment we measured Gemma 3 4B at 32K: llama.cpp allocated 814 MiB, close to the 756 calculated, but vLLM with default settings reserved 2,682 MiB per request, because it sets aside room for the window plus every token it might be processing at once. llama.cpp's --swa-full switch keeps every token in every layer, and with it Gemma's cache went to 4,352 MiB.
• Reuse is harder. Reusing a cached prefix across requests (prefix caching) assumes old tokens are still in the cache. A sliding layer has thrown them away, so runtimes need extra bookkeeping, or keep full caches, to support it.

Attention sinks, and gpt-oss's 128-token window

Xiao et al. (2023) noticed that models put a large share of attention on the first few tokens of a sequence, whatever those tokens say. If a sliding window evicts those first tokens, the model degrades sharply. Keeping the first four tokens' keys and values next to the window fixed it; they called those tokens attention sinks. With plain window attention, the model collapsed as soon as even the first token's key and value were evicted.

gpt-oss builds the idea in: each attention head has a learned sink, a bias in the softmax that lets the head put attention "nowhere" instead of on an arbitrary token. The model card describes it as "a learned bias in the denominator of the softmax, similar to off-by-one attention and attention sinks." You can see it in the checkpoint as a small per-head weight named sinks. OpenAI does not say how much the sink contributes to making a 128-token window work; the StreamingLLM result is a reason to expect a window that short to need something like it.

5. Multi-head latent attention (MLA): store one small vector per token

This is the design from the opening. DeepSeek introduced it in DeepSeek-V2 (2024). Instead of storing keys and values per head, each layer stores one compressed vector per token, the latent, and keeps learned matrices that expand the latent back into all heads' keys and values.

In DeepSeek-V2-Lite the latent has 512 numbers (kv_lora_rank: 512). Position information (RoPE) does not survive that compression trick, for a reason we come to below, so each token also stores a separate 64-number key that carries position (qk_rope_head_dim: 64). That is 576 numbers per token per layer, the same whether the model has 16 heads or 128.

Who uses it. DeepSeek-V2, V3 and V3.2; Moonshot's Kimi K2 (its config declares the DeepseekV3ForCausalLM architecture with the same 512 + 64 dimensions, but 64 heads instead of DeepSeek-V3's 128); and the full-attention layers of Kimi Linear (section 7). This is also why their configs mislead a formula: num_key_value_heads is 16 in V2-Lite and 128 in V3, but those heads exist only after expansion. The field to look for is kv_lora_rank.

What it costs.

The "plain" column counts what the heads would need if each kept its own key (128 + 64 dimensions) and value (128). The more heads a model has, the bigger the saving, because the latent does not grow with heads.

What it buys. A cache smaller than most GQA models of similar size, while every head still gets its own keys after expansion. The DeepSeek-V2 paper reports a 93.3% smaller KV cache and 5.76 times the maximum generation throughput against DeepSeek 67B; that compares two whole models, not MLA against MHA in the same model. Its appendix ablation, on smaller MoE models, finds MLA outperforming MHA. Those are the authors' numbers.

What it gives up.

• Compute moves into decoding. Expanding the latent into keys at every step would be expensive. The standard trick is to fold the expansion matrix into the query projection, so each query is compared with the latents directly; the paper notes that at inference the key and value expansions can be absorbed into the query and output projections. That trick is also why RoPE needs its own separate key: the rotation depends on position, so it cannot be folded into a fixed matrix. The upshot is that MLA decoding does more arithmetic per byte read than GQA. That suits GPUs that have arithmetic to spare, but it needs its own kernels.
• Runtime support. MLA needs dedicated attention kernels. llama.cpp, vLLM and SGLang have them now; smaller runtimes and some quantized-cache paths may not.
• Converting an existing model needs training. A GQA model cannot be switched to MLA by editing its config.

SOTAAZ course

This course alone $79 · 24 lectures · first 3 freeWatch free →

6. Sparse attention: keep every token, read only some

Sparse attention attacks the bandwidth bill, not the memory bill. Every token's cache entry stays in memory; each new token reads only a selected subset.

Who uses it. DeepSeek-V3.2-Exp adds DeepSeek Sparse Attention (DSA) on top of MLA. A small, cheap scorer, the lightning indexer (64 heads of 128 dimensions in the config), rates all earlier tokens for each new token, and attention runs only over the top 2,048 (index_topk: 2048). The report puts it as cutting core attention from O(L²) to O(L·k); the indexer itself is still O(L²) but much cheaper than MLA. DeepSeek's earlier Native Sparse Attention (NSA, 2025) combined three branches: compressed blocks, selected blocks and a local window, gated together (it won a best paper award at ACL 2025).

What it costs. Memory goes slightly up, not down: the full MLA cache is kept, plus the indexer's own key per token. The report makes no memory claim; that the full cache stays is our reading of its equations, since the indexer scores every earlier token. What drops is the attention work per token at long context, from all earlier tokens to 2,048 of them plus a cheap indexer pass over all of them.

What it gives up. The model has to learn which tokens matter, and a token the indexer skips is invisible to that step. It also needs custom kernels, and support outside the major serving engines is thin. Since it does not shrink the cache, it is not a memory answer; it is a speed answer for long inputs.

7. Linear attention and state-space hybrids: most layers keep no tokens

Linear attention removes the per-token cache entirely. Instead of keeping all keys and values, the layer keeps a fixed-size matrix, its state, and folds each new token into it. Reading from the state costs the same at token 100 and token 100,000. Katharopoulos et al. (2020) showed that with this form a transformer layer can run like a recurrent network, with constant memory at inference. Mamba (Gu and Dao, 2023) reached a similar place from state-space models. The newer layers in production models, Gated DeltaNet (Yang et al., ICLR 2025) and Moonshot's Kimi Delta Attention, add a gate that decides how fast the state forgets and a delta rule that overwrites old associations instead of piling new ones on top. Kimi's version extends Gated DeltaNet with finer-grained gating.

The price is that a fixed state cannot hold everything. Ask it to copy a long passage word for word, or retrieve one phone number from a long list, and it has to have compressed exactly the right detail. Studies of copying and associative recall find pure linear and state-space models weaker than attention there. Jelassi et al. (2024) show transformers dramatically outperform state-space models at copying and retrieving from context; the Zoology study (Arora et al., 2023) attributes 82% of the perplexity gap between gated-convolution models and attention to recalling things mentioned earlier in the context. So production models do not go pure linear. They keep a few full-attention layers to do the exact lookups.

Who uses it, and how. Every one of these keeps some full attention:

The ratios come straight from the configs: layer_types or full_attention_interval: 4 for Qwen (one full layer in four), linear_attn_config for Kimi, attn_type_list for MiniMax (0 is lightning, 1 is softmax; the softmax layers sit at every eighth position), and attn_layer_period: 14 with attn_layer_offset: 7 for Jamba (two attention layers out of 28).

What it costs. The cache grows only in the full layers. Qwen3.5-9B's 8 full layers cost 32 KiB per token, 1,024 MiB at 32K, and its 24 linear layers hold a state that does not grow at all. We measured that in llama.cpp: 50.25 MiB of state at 512 tokens, at 4K, at 32K and at 128K. Kimi Linear, with 7 MLA layers, comes to 252 MiB of cache at 32K plus about 43 MiB of state (calculated, not measured).

What it buys. Memory and decode speed that barely depend on length in most layers. The Qwen team's claim for Qwen3-Next is more than ten times the throughput of Qwen3-32B for contexts over 32K tokens; Moonshot reports up to 75% less KV cache and up to six times faster decoding at 1M tokens for Kimi Linear, against full MLA trained the same way. Those are the vendors' numbers on their own setups.

What it gives up.

• Exact recall lives in a few layers. The hybrid ratio is the dial between memory and recall, and the vendors set it differently: three linear layers for every full one in Qwen and Kimi (Kimi's 27 layers come out at 20 and 7), one in eight for MiniMax, one in fourteen for Jamba Reasoning 3B.
• Features built for caches get harder. Prefix caching, rolling back a rejected speculative draft, and branching a conversation all assume you can cut the cache at any token. A state cannot be cut; it has to be snapshotted. In our vLLM measurements, Qwen3.5-9B's fixed memory per request doubled, from 3 blocks to 6, when prefix caching was on; vLLM's log shows it switching the state cache to an "align" mode in that case. We did not confirm in the source why that mode needs the extra blocks.
• Runtime support is newer. Gated DeltaNet and its relatives need their own kernels for both prefill and decode. The major engines have them for the popular models; ports and quantized paths lag.

The seven, side by side

The designs stack. gpt-oss is GQA and sliding window. Kimi Linear is linear attention with MLA for its full layers. DeepSeek-V3.2 is MLA with sparse reads. When you read a new model card, the useful question is not "which one does it use" but "which of the five things does each layer cut."

What we did not check

• Quality and speed claims are the authors'. We did not measure any model's accuracy, recall or decode speed for this post. Where a paper or vendor reports a quality or throughput number, we say whose it is.
• Memory numbers are calculated from config.json, at 32K, in 16-bit precision. Runtimes add their own reserves. For six of these models we measured the real allocation in llama.cpp and vLLM, and the gaps were large in places; that is a later post. The state sizes for Kimi Linear are calculated only; Qwen3.5-9B's was measured.
• The model list is not complete. We picked models whose configs we could read (three via Unsloth's copy, as noted above) and that show each design clearly. There are other variants we left out: cross-layer KV sharing, KV cache quantization, and multi-token latent designs, among others.

Next

The rest of the series goes from here to your GPU:

• Seven attention designs in PyTorch: each design above as a short function, with a check that it does what this post says, including MLA's weight absorption on DeepSeek-V2-Lite's real weights.
• How six models store the KV cache, measured: one or two models per design, what llama.cpp and vLLM allocate at 4K, 32K and 128K.
• What changes when you deploy on llama.cpp vs vLLM: why the same model at the same length took three times the memory in one runtime, and the settings that move it.