Attention Sink: Why Qwen3-8B Gives Up to 77% of Its Attention to the First Word, and Why 'Banana' Works Just as Well

Attention Sink: Why Qwen3-8B Gives Up to 77% of Its Attention to the First Word, and Why 'Banana' Works Just as Well#

We fed Qwen3-8B 20 stretches of Wikipedia text and asked one question: when a word in the middle of a passage looks back over the words before it, where does it look? In layer 29, the answer was mostly "at the very first word". That one word got 77% of the attention, while the 256 to 511 other words it could see split the rest. The first word was not special. In one passage it was " Robert", in another "oul", the tail end of a word cut in half. When we replaced it with " banana", the share barely moved (0.4588 vs 0.4577, averaged over layers 2-35).

This has a name, attention sink, and researchers have been studying it since 2023. This post has three parts: what we measured on Qwen3-8B and why we set it up that way, five papers that explain why the sink forms and what it does, and a notebook so you can run the same measurement yourself.

First, what "attention" means here#

When a language model reads a word, each layer lets that word look back at the earlier words and decide how much to take from each. The "how much" is a list of weights, one per earlier word, and the weights always add up to 1. That last detail is the whole story of this post.

A layer does this many times in parallel. Each parallel copy is called a head, and Qwen3-8B has 32 heads in each of its 36 layers. Each head makes its own list of weights.

So when we say "77% of the attention goes to the first word", we mean this: in each of the 20 chunks, take the words at positions 256-511, take all 32 heads, and average the weight each of them puts on position 0. In layer 29 that average is 0.77.

Why we ran this, and how#

The attention sink was first described on 2023 models such as Llama 2. We wanted to see it with our own eyes on a model people run today, and we had three questions.

1. Is it there in Qwen3-8B, and how strong is it? We were not sure. Attention scores how related two words are by comparing two vectors, a query for the word that is reading and a key for each earlier word. Qwen3 normalizes both before comparing them (QK-norm), which could make it harder for one position to grab most of the attention.
2. Is it about the position or the word? If the sink is a word the model finds meaningful, swapping that word should change it. If it is the position, the word should not matter.
3. Does the model need to see that token? If we hide it, does the model get worse at predicting the next word?

The setup, with the reason for each choice:

• Qwen3-8B, bf16. A current open model that fits on one GPU. It also adds no special start token to the text, so position 0 is an ordinary word piece. That makes question 2 a clean test. In models that always put a special start token first, you cannot separate "the first position" from "that special token".
• WikiText-2 test set, 20 chunks of 1,024 tokens. Ordinary encyclopedia text that anyone can download. We dropped blank lines and section headings and cut the rest into non-overlapping chunks, so each chunk starts at a random point in the text.
• Words at positions 256-511 of each chunk. Every one of them has at least 256 earlier words to choose from. If it spread its attention evenly, the first word would get only about 0.27%. That makes a large share on position 0 easy to tell apart from chance.
• "Eager" attention in transformers. The fast attention kernels never write the weights out. The plain implementation does, when asked with output_attentions=True.

The core of the measurement is three lines:

att = model(input_ids, output_attentions=True).attentions   # one tensor per layer
# att[layer] has shape [batch, head, query, key]; key 0 is the first token
share = [a[0, :, 256:512, 0].mean().item() for a in att]     # per layer: average over heads and queries

For question 3 we pass our own attention mask. A normal mask lets each word see itself and every earlier word. To hide position 0, we also block it for every word after it. The token stays in the text and keeps its position number; no later word can look at it.

m = torch.full((n, n), NEG).triu(1)   # NEG = most negative bf16 number: "cannot see"
m[1:, 0] = NEG                        # nobody after position 0 may look at position 0

Before trusting that, we checked that passing a plain causal mask (each word sees itself and earlier words, nothing more) gives exactly the same output as passing no mask. It did (largest difference 0.0).

We wrote our predictions and this method down, in a timestamped record, before running anything. One prediction was: in at least 30 of the 36 layers, the first token would receive at least 20% of the attention. In plain terms: of the total weight of 1 that a word hands out, 0.2 or more would go to position 0, averaged over heads and over positions 256-511. 20% is about 74 times the even split (0.27%), far too much to be chance. We put our confidence at about 65%, because of the QK-norm doubt above.

There were eight predictions in all; we score every one below, hits and misses. You can rerun all three measurements in the notebook that comes with this post (see the end).

What we measured#

• Layers 0-6: almost nothing. The first word gets 0.04% to 1.9%.
• Layer 7: it jumps to 62%.
• Layers 7-35: between 26% and 77% in every layer. The highest is layer 29 at 77%.

So 29 layers cleared 20%, one short of our 30. The prediction was wrong by one layer, and we count it as wrong. The pattern itself was stronger than we expected: in layer 29 the first word gets about 285 times the even split (0.7714 / 0.00270, our calculation).

Then we swapped the first token. We kept the rest of each chunk the same and replaced its first token with " the", ".", a newline, or " banana". Averaged over layers 2-35:

The word did not matter. The position did. That was our prediction too.

Why would a model hand most of its attention to whatever happens to come first? Five papers answer that one step at a time.

Step 1: softmax has to put the attention somewhere#

Paper: Xiao et al., "Efficient Streaming Language Models with Attention Sinks" (MIT, Meta AI, CMU, NVIDIA; ICLR 2024, arXiv 2309.17453). This is the paper that named the attention sink.

What they asked. Why do so many heads in Llama 2 and similar models put a large weight on the first few tokens, even when those tokens have nothing to do with the current word?

Their explanation. The weights come out of softmax, the function that turns raw scores into shares that add up to 1. Softmax has no way to say "none of these words is useful, give zero to all of them". In their words: "even when the current query does not have a strong match in many previous tokens, the model still needs to allocate these unneeded attention values somewhere so it sums up to one."

That is an explanation, not yet a test. The test came a year later.

Paper: Gu et al., "When Attention Sink Emerges in Language Models: An Empirical View" (Sea AI Lab and NUS; ICLR 2025, arXiv 2410.10781).

What they asked. What in training makes the sink appear: the optimizer, the data, the loss, or the attention itself?

How they tested. First they measured existing models: GPT-2, OPT, Pythia, Llama 2, Llama 3 and Mistral. Their measure counts the share of heads that put more than 30% of their attention on the first token. They found the sink in all of these families, even in Pythia-14M, a model with only 14 million parameters. Then they trained small Llama-style models from scratch (about 60 million parameters, larger runs up to 1 billion) and changed one thing at a time.

What they found. The sink appears only after the model has trained well on enough data; cut the training data from 5 billion tokens to 50 million and it does not form. The decisive run swapped softmax for a sigmoid, which scores each word on its own without forcing the scores to add up to 1. In their 1B model the sink measure fell from 45.11% with softmax to 2.46% with the plain sigmoid, while validation loss stayed close (3.07 vs 3.10). The 2.46% is measured on stand-in scores, because sigmoid weights are not shares. When they added the add-up-to-1 normalization back to sigmoid, the sink came back.

What it means for our result. In Gu et al.'s account, the sink comes from the add-up-to-1 rule plus enough training. Qwen3-8B has both, and it has the sink. Our result is consistent with their explanation.

Step 2: why the first word#

If the model needs a place to put spare attention, it needs a place every word can reach. In a model that reads left to right, each word sees only the words before it. The first word is the only one every later word can see. Xiao et al. again: "initial tokens are visible to almost all subsequent tokens ... making them more readily trained to serve as attention sinks."

Their paper also has an early version of our banana test. In Llama-2-13B they replaced the first four tokens with four newlines. Perplexity, a standard measure of how surprised the model is by the text (lower is better), was 5.60, against 5.40 with the original tokens, on one book from the PG19 collection, with the model keeping only the first 4 tokens plus the most recent 1,020. Keeping only recent tokens gave 5,158.07, so what mattered was keeping the first position, whatever sat there. Their conclusion: "the absolute position of the starting tokens, rather than their semantic value, holds greater significance."

What makes that position pull attention so hard? A third paper looked inside the numbers.

Paper: Sun et al., "Massive Activations in Large Language Models" (CMU, Meta AI and Bosch Center for AI; arXiv 2402.17762).

What they found. Among the thousands of numbers that represent each word inside the model, a handful are enormous. In Llama2-7B the largest is about 2,000, around 10,000 times the typical (median) value of about 0.2. They found the same in Llama2-13B, Mixtral-8x7B and Phi-2. These massive activations sit on the starting word and, in some models, on the first period or newline. They "largely stay constant regardless of the input", so the authors describe them as fixed biases: the model has learned to keep a constant there.

How they tested the bias idea. In Llama2-7B they set just those few values to zero, and perplexity went to infinity on WikiText, C4 and PG-19. The model broke completely. Then they set the same values to their average instead, and perplexity did not move (5.47 on WikiText both times). The model needs the value to be there. It does not need the value to depend on the text.

They also connected these values to attention: from layer 3 on in Llama2-7B, attention is "mostly concentrated on the two tokens associated with massive activations".

Gu et al. reach the same picture from the attention side: the sink "acts as key biases, storing extra attention scores, which could be completely non-informative and not contribute to the value computation." Put simply, position 0 becomes a fixed slot that soaks up attention and gives back almost nothing.

Step 3: what the sink is for#

A slot for spare attention explains where it goes. It does not explain why the model would want spare attention at all.

Paper: Barbero et al., "Why do LLMs attend to the first token?" (Oxford, Google DeepMind and others; COLM 2025, arXiv 2504.02732).

Their answer. At every layer, attention mixes information between words. In a deep model reading a long text, too much mixing, layer after layer, makes every word's representation look alike, and the model loses track of which word is which. The authors call this over-mixing. A head that sends its attention to the first word adds almost nothing to the word it is processing, because the value stored there is small (they show this for one Gemma 7B head). In the paper's words, attention sinks "help construct approximate no-ops". A head gets a way to sit out a turn.

What they measured. Gemma 7B puts a special start token (⟨bos⟩) at position 0. When they removed it at inference time, ARC-e accuracy fell from 80.77 to 28.49, HellaSwag from 80.61 to 27.35, and the RULER long-context score at 4,096 tokens from 82.57 to 0.00. The sink also grows with model size in the Llama 3.1 family: by their measure, 45.97 for 8B, 73.49 for 70B and 78.29 for 405B. The authors read this as bigger, deeper models needing more heads that can sit out.

The experiment that matches ours. They also trained small models themselves. Models trained with ⟨bos⟩ always at the front relied on it: remove ⟨bos⟩ and the sink went away. Models trained without ⟨bos⟩ formed the sink "at the first token regardless, but is slightly weaker". Qwen3-8B adds no start token, and its sink sat on whatever word came first. That is the same behavior.

SOTAAZ course

This course alone $49 · 28 lectures · first 3 freeWatch free →

Step 4: where it breaks things#

The sink became well known because of a practical problem. To keep a long chat going with limited memory, a common trick is to keep only the most recent tokens in the KV cache, the stored keys and values of earlier words, and drop the oldest as new ones arrive. Sooner or later that drops the first token, and the sink goes with it.

Xiao et al. measured what happens on 400,000 tokens of PG19 text. With a window of the last 4,096 tokens, Llama-2-7B's perplexity was 3,359.95, which means the output was garbage. Keeping the first 4 tokens plus the last 4,092 brought it down to 9.59. One kept token gave 11.88 and two gave 10.51; they conclude that "introducing four initial tokens generally suffices". MPT-7B, with a 2,048-token cache, went from 460.29 to 14.99 the same way, and Falcon-7B and Pythia-12B improved too.

Their fix, StreamingLLM, is exactly that: keep a few sink tokens forever and slide the window over the rest. They report stable output over 4 million tokens and up to 22.2 times the speed of the alternative that recomputes the window from scratch. They also trained a model with one extra learnable token at the start of every training sample, a dedicated sink, and that model needed only this one token to stream without trouble.

Step 5: designing it down#

If the sink exists because attention has no way to say "nothing here", the fix is to give it one.

Paper: Qiu et al., "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free" (Qwen Team, Alibaba Group, with university co-authors; arXiv 2505.06708).

What they changed. They add a small sigmoid gate after the attention output, so each head can turn its own output down toward zero. A head with nothing useful to add no longer has to park its attention somewhere. It can just close the gate. They compared more than 30 variants, training 15B mixture-of-experts models and 1.7B dense models on up to 3.5 trillion tokens.

What they found. In the model they analyze, averaged over layers, a gate that scales each element of the attention output separately (so every head gets its own gates) cut the share of attention on the first token from 46.7% to 4.8%. A gate with a single value per head gave 7.3%. Sharing one gate across heads (30.1%) or gating the values instead (29.7%) helped much less. In one layer it fell from 83% to 4%. The largest activations dropped too, from an average of 1,053 to 94. One finding is a warning against the neat story of Step 2: "massive activations are not a prerequisite for attention sinks". Some gate variants cut the activations a lot and the sink much less.

Qwen3-8B, the model we measured, has no such gate after attention, and it has the sink.

Will you see this in other models?#

We measured only Qwen3-8B. But every model family without an output gate that these papers measured showed the sink: GPT-2, OPT, Pythia (down to 14 million parameters), Llama 2, Llama 3 and 3.1, Mistral, MPT, Falcon, Phi-2 and Gemma. All of them read left to right and use standard softmax attention with no gate after it. Based on these papers, a model built that way will likely show a similar pull toward the first position. Gu et al. found it in chat models as strongly as in their base models, and even when the input was random tokens.

What changes, even among models built that way, and the known exceptions:

• Where it sits. Models that always put a special start token first, such as Gemma or Llama, put the sink on that token. Models without one, like Qwen3, put it on whatever comes first. In small models Barbero et al. trained themselves, the ones trained without a start token had a slightly weaker sink.
• How strong, and in which layers. In Qwen3-8B the first 7 layers have almost none. Other models will have their own pattern.
• Layers that only see the last few thousand tokens (sliding window). Gu et al. found that once the true first token is out of view, the earliest token left in the window does not become a new sink (Mistral-7B too), and that small windows keep the sink from forming.
• Models with a dedicated sink slot. With a learnable key bias or a learnable sink token at the front, the sink moves there instead of the first word (Gu et al.).
• Gated models. Models with a gate on each head's attention output, per element or per head as in Step 5, should show much less.
• The same token repeated. Gu et al. found that on such input the sink disappears in Mistral and Llama models.

The notebook takes a model name as a setting. It is written for tokenizers that, like Qwen3's, add no start token: with Llama or Gemma, measurement 1 runs but measurement 2 stops.

One thing we tried that the papers do not cover#

For question 3 we hid the first token from every later word, with everything else in place, and measured the average next-word loss (lower is better) for positions 512-1023 on the same 20 chunks.

The five values are at most 0.0033 apart. We had predicted that hiding the first token would add at least 0.5, so that prediction was wrong. We also predicted that hiding the first four would hurt at least as much as hiding one, and that was wrong too (2.1358 is below 2.1370). We predicted the token-256 control would move by less than 0.05, and it did.

We also had the model write. We gave it 20 requests in chat format and let it write 64 tokens each, always picking the most likely next token. Hiding token 0 (the chat format's start marker) made 15 of 20 answers drift from the original somewhere. Hiding token 2, a newline, as a control made 14 drift. In both conditions, every drifted answer, up to its end marker, reads as a normal sentence, just worded differently. We also counted repeated phrases; most of the difference came from text written after the answer's end marker. In a count we added after seeing the results, counting only up to that marker, nothing hidden gave 0.0049, and hiding token 0 and hiding token 2 both gave 0.0082. So we cannot say hiding the first token unsettles the writing; it drifts about as much as hiding any one token.

This does not contradict Step 3 or Step 4. The Step 4 window drops thousands of early tokens at once; we hid one, or four. Step 3 removed ⟨bos⟩ from the input, a token Gemma saw at the front of every training sample; we kept the token and only masked it. In the loss test, Qwen3-8B has no start token, so our position 0 was an ordinary word piece. In the writing test, token 0 was the chat start marker, closer to Step 3's ⟨bos⟩; we did not measure how much attention it got, and hiding it moved the answers about as much as hiding a newline. We did not measure where the attention went once position 0 was hidden, so we do not know why nothing changed.

Eight predictions, scored#

Try it yourself#

The notebook is in the zip under "Files for this post" at the end of this page, together with our measurement script, the predictions we wrote before running, and every number in this post (results.json).

The notebook runs all three measurements with the settings above and prints our numbers next to yours. It leaves out the writing test. Qwen3-8B in bf16 is about 16.4 GB in weights alone, plus some room for the attention weights; we ran it on one 80 GB A100. On a smaller card you can set the model name to Qwen3-0.6B or Qwen3-1.7B, but we have not run those two.

Limits#

• Our numbers come from one model (Qwen3-8B, bf16, transformers 4.56.2) and one text source (20 chunks of WikiText-2), with attention shares taken at positions 256-511.
• The shares are averages over 32 heads. Individual heads vary far more than the averages show.
• All paper numbers are the papers' own, for the models and data they name. We did not rerun them.
• We measured the attention shares with the plain (eager) implementation from the start. For question 3 and the writing test we had planned a faster one, but it failed the mask check above, with a largest difference of 9.05 instead of 0.0. Before seeing any of those results, we recorded the switch to eager and ran everything again. The share on the first word came out the same both times.

Next in this series we open another part of the model and check our guess against what is inside. If you want it in your inbox, the newsletter is below.