Models & Algorithms•SOTAAZ Lab••KR

The Transformer Block from 2017 to Mamba Hybrids: 15 Changes, 13 Rebuilt in Code

Fifteen changes to the Transformer block, from Post-LN to Mamba hybrids, each drawn as one diagram and explained with what the paper reported and what it costs. We rebuilt one layer from seven real models as short PyTorch functions; all seven match the Hugging Face output. On the same sentence, the residual stream grows 98-fold from the first layer to the last in Qwen3-0.6B and only 1.1-fold in BERT.

The Transformer Block from 2017 to Mamba Hybrids: 15 Changes, 13 Rebuilt in Code

We fed one sentence to seven models, BERT, GPT-2, Pythia, Qwen3, Gemma 3, Jamba and Granite-4.0-H, and measured how large the last token's vector gets after each layer. In Qwen3-0.6B-Base it ends 98 times larger than after the first layer. In BERT it ends 1.1 times larger, almost the same. Both are built from the same three cells, attention, a feed-forward network and a norm, though they differ in many other ways too. One difference shows up directly in this chart: BERT renormalizes the residual stream itself after every addition, and the other six never normalize it. That is why only BERT's line stays flat. Where the norm sits is one of the fifteen changes this post walks through.

How much the residual stream grows from the first layer to the last, for the same sentence in seven models. The x axis is depth as a fraction of the layer count; the y axis, on a log scale, is the vector size divided by its size after layer 1. Qwen3-0.6B-Base ends at about 98 times, Gemma-3-1B-it at 47, Jamba-tiny-dev at 31, Pythia-70m at 14, GPT-2 at 9.9, Granite-4.0-H-350M at 9.3. BERT-base, the only Post-LN model, stays between 0.56 and 1.1 times throughout.

In the previous post we followed one token through GPT-2, Qwen3 and Granite. Every layer had the same steps: normalize, mix information across tokens, add the result back, normalize, transform each token with a feed-forward network (FFN), add back. This post is about how those cells have been filled, wired and stacked since 2017. For each change there is one animated diagram, what the paper reported, and what it costs. The attention calculation itself (heads, KV cache, grouped queries, sparse and linear attention) is covered in the attention roundup, so here attention is a single cell.

How we checked the code

Every block in this post comes with a short PyTorch function. To know that a function is the real block and not our reading of a paper, we compared it with the actual model:

  1. Load the model from Hugging Face and run the sentence "In the morning, Alice gave Bob a book. Later, Bob gave the book back to" through it.
  2. While it runs, record what goes into and comes out of each layer. PyTorch calls this a forward hook.
  3. Take one layer's weights, give our function the same input, and compare its output with what the real layer produced.
python
io = {}
def hook(i):
    def f(mod, args, kwargs, out):
        x = args[0] if args else kwargs["hidden_states"]   # what went into layer i
        y = out[0] if isinstance(out, tuple) else out       # what came out
        io[i] = (x[0].clone(), y[0].clone())
    return f
hs = [l.register_forward_hook(hook(i), with_kwargs=True) for i, l in enumerate(layers)]
model(ids)

x, y = io[5]                                     # layer 5
ours = pre_ln_block(x, weights_of_layer_5, n_heads, eps)
print((ours - y).abs().max())                    # largest difference over all numbers

We ran this on a CPU in float32 for 22 layers across seven models (the full script is in the repro bundle at the end). The largest difference, divided by the largest value in the real output, was 3.9e-5, in Pythia's last layer. Every other check was 2.2e-7 or less. Pythia's last layer stands out, so we ran the three Pythia checks again in float64: the gap in that layer fell to 5.4e-13, and in the other two to 2.3e-15 or less. A gap that shrinks with the precision is rounding, not a different calculation, so the functions compute the same thing as the models.

ModelWhat it checksLayers comparedLargest relative difference
BERT-basePost-LN0, 5, 112.2e-7
GPT-2Pre-LN0, 5, 111.8e-7
Pythia-70mparallel block, partial RoPE0, 2, 53.9e-5
Qwen3-0.6B-BaseRMSNorm, QK-norm, RoPE, SwiGLU0, 14, 271.4e-7
Gemma-3-1B-itnorm before and after each cell2, 5, 254.5e-8
Jamba-tiny-devMamba, MoE, attention without positions3, 4, 5, 129.4e-8
Granite-4.0-H-350MMamba-2, attention without positions0, 10, 311.6e-7

The code shown below is the code that passed these checks. Two items, the encoder-decoder original and Hymba, are explained with diagrams and the papers' numbers.

Where it started

1. The original block (2017)

The original Transformer block. Left, an encoder layer: self-attention, add to x, LayerNorm; FFN with ReLU, add, LayerNorm. Right, a decoder layer with a third sub-layer, cross-attention to the encoder, between masked self-attention and the FFN. In both, LayerNorm (LN) sits on the main stream after each addition.

"Attention Is All You Need" (Vaswani et al., 2017) built a translation model from an encoder and a decoder, six layers each. Each sub-layer is wrapped as LayerNorm(x + Sublayer(x)): compute the cell, add its output to x, then normalize the sum. This is called Post-LN, because the norm comes after the addition. The FFN is two matrices with a ReLU between them, 512 wide outside and 2,048 inside. Positions were added to the input as fixed sine waves; the paper also tried learned position vectors and reports the same BLEU, 25.7, for both (Table 3, row E).

BERT kept this block for its encoder (with GELU in the FFN), so BERT-base is our working Post-LN example:

python
def post_ln_block(x, sd, H, eps):
    q = heads(x @ sd["attention.self.query.weight"].T + sd["attention.self.query.bias"], H)
    k = heads(x @ sd["attention.self.key.weight"].T + sd["attention.self.key.bias"], H)
    v = heads(x @ sd["attention.self.value.weight"].T + sd["attention.self.value.bias"], H)
    a = merge(attend(q, k, v, 1 / math.sqrt(q.shape[-1]), causal=False))
    a = a @ sd["attention.output.dense.weight"].T + sd["attention.output.dense.bias"]
    x = layer_norm(x + a, sd["attention.output.LayerNorm.weight"], sd["attention.output.LayerNorm.bias"], eps)
    f = F.gelu(x @ sd["intermediate.dense.weight"].T + sd["intermediate.dense.bias"])
    f = f @ sd["output.dense.weight"].T + sd["output.dense.bias"]
    return layer_norm(x + f, sd["output.LayerNorm.weight"], sd["output.LayerNorm.bias"], eps)

Look at the two layer_norm(x + ...) lines: the stream x itself is renormalized twice per layer. That is why BERT's line in the first chart stays flat.

2. Decoder only

Left, the original decoder layer with masked self-attention, cross-attention and FFN. Right, the GPT layer from 2018 with only masked self-attention and FFN with GELU.

A language model that only continues text has no source sentence to read, so it does not need the encoder or the cross-attention that reads from it. GPT (Radford et al., 2018) stacked 12 decoder layers with masked self-attention and an FFN: 768 wide, 3,072 inside. The same paper switched the FFN activation to GELU and used learned position vectors instead of sine waves. Almost every model below is this decoder-only stack with different cells.

The norm cell

3. Pre-LN: normalize the copy, not the stream

Left, Post-LN: LN on the main stream after each addition. Right, Pre-LN as in GPT-2: LayerNorm at the start of each branch, before attention and before the FFN, and nothing on the main stream.

GPT-2 (Radford et al., 2019) states three changes from GPT: LayerNorm "was moved to the input of each sub-block", one more LayerNorm was added after the last block, and residual weights were scaled down by 1/√N at initialization. In code the difference from BERT is where the norm call goes:

python
def pre_ln_block(x, sd, H, eps):
    D = x.shape[-1]
    h = layer_norm(x, sd["ln_1.weight"], sd["ln_1.bias"], eps)      # normalize a copy
    q, k, v = (heads(t, H) for t in (h @ sd["attn.c_attn.weight"] + sd["attn.c_attn.bias"]).split(D, -1))
    a = merge(attend(q, k, v, 1 / math.sqrt(D // H)))
    x = x + a @ sd["attn.c_proj.weight"] + sd["attn.c_proj.bias"]   # add to the raw stream
    h = layer_norm(x, sd["ln_2.weight"], sd["ln_2.bias"], eps)
    f = F.gelu(h @ sd["mlp.c_fc.weight"] + sd["mlp.c_fc.bias"], approximate="tanh")
    return x + f @ sd["mlp.c_proj.weight"] + sd["mlp.c_proj.bias"]

Why it helps. Xiong et al. (2020) showed that at initialization a Post-LN model has large gradients near the output, so it needs a learning-rate warm-up to train well; on IWSLT14 German-English, Post-LN without warm-up reached only 8.45 BLEU against about 34 with it. Pre-LN gradients are well-behaved, so warm-up can be dropped. In their BERT pre-training run, Pre-LN reached a validation loss of about 1.69 near 500k steps and Post-LN only near 700k (their Figure 5a), which they report as a 40% speed-up.

What it costs. The stream is never normalized inside the stack, so every layer's output accumulates. That is the growth in the first chart: GPT-2's last-token vector goes from 51.6 after layer 1 to 512.1 after layer 12. A final norm before the output (the extra LayerNorm GPT-2 added) handles the scale at the end.

4. RMSNorm: drop the mean

Left, LayerNorm in the Pre-LN position of GPT-2. Right, the same block with RMSNorm, as in Llama and Qwen3.

LayerNorm subtracts the mean of the vector, divides by the standard deviation, then multiplies by a weight and adds a bias. RMSNorm (Zhang and Sennrich, 2019) keeps only the scaling: divide by the root mean square and multiply by a weight.

python
def rms_norm(x, w, eps, plus_one=False):
    y = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps)
    return y * (1 + w) if plus_one else y * w           # Gemma multiplies by (1 + w)

What the paper reports. On a Transformer-base translation model the time per 1,000 training steps fell from 248 to 231 seconds, 6.9% (Table 4), with BLEU 26.6 vs 26.8 on newstest2014. The abstract's "7% to 64%" range includes RNN models; for the Transformer the reported gains are 6.9% for RMSNorm and 9% for its partial variant, pRMSNorm. Cost: none stated beyond losing re-centering, which the authors argue is not needed.

5. A norm on both sides of each cell (Gemma 2 and 3)

Left, a Pre-LN block as in Qwen3. Right, Gemma 2 and 3: RMSNorm before attention and again on the attention output before it is added; the same before and after the FFN.

Gemma 2 (Gemma Team, 2024) normalizes "the input and output of each transformer sub-layer"; Gemma 3 (Gemma Team, 2025) keeps it. Each Gemma 3 layer therefore has four norms: input_layernorm, post_attention_layernorm, pre_feedforward_layernorm, post_feedforward_layernorm.

python
def sandwich_block(x, sd, Hq, Hkv, d, eps, theta, scale):
    pos = torch.arange(x.shape[0])
    n = lambda t, name: rms_norm(t, sd[name], eps, plus_one=True)
    h = n(x, "input_layernorm.weight")
    q = n(heads(h @ sd["self_attn.q_proj.weight"].T, Hq), "self_attn.q_norm.weight")
    k = n(heads(h @ sd["self_attn.k_proj.weight"].T, Hkv), "self_attn.k_norm.weight")
    v = heads(h @ sd["self_attn.v_proj.weight"].T, Hkv)
    q, k = rope(q, pos, theta), rope(k, pos, theta)
    a = merge(attend(q, k, v, scale)) @ sd["self_attn.o_proj.weight"].T
    x = x + n(a, "post_attention_layernorm.weight")                  # norm the output, then add
    h = n(x, "pre_feedforward_layernorm.weight")
    g, u = h @ sd["mlp.gate_proj.weight"].T, h @ sd["mlp.up_proj.weight"].T
    f = (F.gelu(g, approximate="tanh") * u) @ sd["mlp.down_proj.weight"].T
    return x + n(f, "post_feedforward_layernorm.weight")

A naming trap if you read the weights: in Llama and Qwen3, post_attention_layernorm is the norm before the FFN. In Gemma it really is applied to the attention output.

The output norm limits how much each cell can add, but the stream itself is still never normalized, so it still grows: 47-fold in our chart. That is more than GPT-2's 9.9-fold, but the two models differ in size and training too, so the chart cannot isolate the effect of the output norm. What the papers report: the reason given is "to stabilize training"; neither report gives a number for this change. Cost: two more norm calls per layer.

6. QK-norm: normalize queries and keys

Left, an attention branch as in GPT-2: norm, Q/K/V projections, scores q·k, weighted sum of v. Right, the same with an RMSNorm on q and k, per head, between the projections and the scores, as in Qwen3 and Gemma 3.

Attention scores are dot products of queries and keys. If those vectors grow during training, the scores grow and softmax puts nearly all weight on one token. ViT-22B (Dehghani et al., 2023) hit exactly this at about 8B parameters: training diverged because attention logits became extremely large. In their runs without the normalization, logits grew "to over 50000 in magnitude". Their fix, which they credit to Gilmer et al. (2023), was a LayerNorm on the queries and keys before the dot product. An earlier paper (Henry et al., 2020) used L2 normalization with a learned scale and reported 0.928 BLEU better on average over five low-resource translation pairs.

Qwen3 (Yang et al., 2025) removed the QKV bias of Qwen2 and added QK-norm "to ensure stable training"; Gemma 3 replaced Gemma 2's logit soft-capping with it. In Qwen3 it is two extra lines, one RMSNorm per head on q and on k:

python
def qwen3_block(x, sd, Hq, Hkv, d, eps, theta):
    pos = torch.arange(x.shape[0])
    h = rms_norm(x, sd["input_layernorm.weight"], eps)
    q = rms_norm(heads(h @ sd["self_attn.q_proj.weight"].T, Hq), sd["self_attn.q_norm.weight"], eps)
    k = rms_norm(heads(h @ sd["self_attn.k_proj.weight"].T, Hkv), sd["self_attn.k_norm.weight"], eps)
    v = heads(h @ sd["self_attn.v_proj.weight"].T, Hkv)
    q, k = rope(q, pos, theta), rope(k, pos, theta)
    x = x + merge(attend(q, k, v, 1 / math.sqrt(d))) @ sd["self_attn.o_proj.weight"].T
    return x + swiglu(rms_norm(x, sd["post_attention_layernorm.weight"], eps), sd)

This one function also contains items 4, 7 and 10. Cost: a small weight vector per norm (128 numbers each in Qwen3-0.6B) and two norm calls per layer.

The position cell

7. RoPE: position as a rotation inside attention

Left, GPT-2: a learned position row added to the token row once, before the first layer. Right, Qwen3: only the token row goes in, and q and k are rotated by position inside every attention layer.

GPT-1 and GPT-2 add a learned position vector to each token vector once, at the bottom; GPT-2 has 1,024 such rows and cannot take a longer input. RoPE (Su et al., 2021) instead rotates pairs of numbers in q and k by an angle proportional to the position, inside every attention layer. The dot product of two rotated vectors then depends on how far apart the tokens are. The previous post has the rope function and a check of that property.

What the paper reports: 27.5 vs 27.3 BLEU on WMT14 English-German against the Transformer base, and on a Chinese legal-matching task, 68.29% accuracy at 512 tokens and 69.79% at 1,024 against 67.77% for the BERT baseline. Limitation stated by the authors: they lack "thorough explanations on why it converges faster". Pythia applies RoPE to only the first quarter of each head (rot_frac = 0.25 in our parallel_block below); Qwen3-Next does the same in its attention layers.

8. NoPE: no position signal at all

Left, Qwen3 attention with q and k rotated by position. Right, Granite-4.0-H and Jamba: the same attention with no rotation and no position row; only the causal mask is left.

A decoder already has some position information without any encoding: the causal mask lets token 10 see 10 earlier tokens and token 3 see only 3. Kazemnejad et al. (2023) trained small decoder-only models (about 107M parameters) on synthetic reasoning tasks and found that no position encoding generalized to longer inputs better than the explicit methods they tested. Limitation stated: they could not test large-scale pretraining, and in their 1B runs ALiBi was more stable than NoPE at the longest lengths.

The hybrids use NoPE for a different reason. Jamba (Lieber et al., 2024): "with the Mamba layer, positional embeddings or mechanisms like RoPE are not necessary". Granite-4.0-H-350M's config says "position_embedding_type": "nope". A Mamba layer reads tokens in order, so the order is already in its state. Our Jamba and Granite attention code has no rope call, and both matched the real layers.

The wiring

9. Parallel block: attention and FFN read the same input

Left, the sequential GPT-2 block: attention is added to x, then the FFN reads the updated x. Right, the parallel block as Pythia builds it: both branches read the same x, each through its own norm, and both outputs are added in one step. GPT-J and PaLM use the same wiring with one shared norm.

In a normal block the FFN sees x after attention has been added. GPT-J (Wang and Komatsuzaki, 2021) computed both from the same x and added them together. PaLM (Chowdhery et al., 2022) adopted it and reported "roughly 15% faster training speed at large scales, since the MLP and Attention input matrix multiplications can be fused", along with "a small quality degradation at 8B scale but no quality degradation at 62B scale", from which they extrapolated that it would be neutral at 540B. GPT-NeoX and Pythia use it too:

python
def parallel_block(x, sd, H, eps, theta, rot_frac):
    T, D = x.shape
    pos = torch.arange(T)
    h = layer_norm(x, sd["input_layernorm.weight"], sd["input_layernorm.bias"], eps)
    qkv = (h @ sd["attention.query_key_value.weight"].T + sd["attention.query_key_value.bias"])
    qkv = qkv.view(T, H, 3 * (D // H)).transpose(0, 1)              # [q|k|v] per head
    q, k, v = qkv.chunk(3, -1)
    rd = int((D // H) * rot_frac)                                   # Pythia rotates 1/4 of each head
    q, k = rope(q, pos, theta, rd), rope(k, pos, theta, rd)
    a = merge(attend(q, k, v, 1 / math.sqrt(D // H)))
    a = a @ sd["attention.dense.weight"].T + sd["attention.dense.bias"]
    h2 = layer_norm(x, sd["post_attention_layernorm.weight"], sd["post_attention_layernorm.bias"], eps)
    f = F.gelu(h2 @ sd["mlp.dense_h_to_4h.weight"].T + sd["mlp.dense_h_to_4h.bias"])
    f = f @ sd["mlp.dense_4h_to_h.weight"].T + sd["mlp.dense_4h_to_h.bias"]
    return x + a + f                                                # one addition for both

Pythia has two separate norms (h and h2), not one shared norm. The GPT-NeoX paper (Black et al., 2022) says this was unintentional and that small-scale experiments showed no difference. Cost: inside one layer, the FFN can no longer use what attention just found.

The FFN cell

10. SwiGLU: a gated FFN

Left, GPT-2's FFN: up from 768 to 3,072, GELU, down to 768. Right, Qwen3's SwiGLU: two projections, gate and up; SiLU of the gate multiplies the up projection element by element; then down.

The FFN in GPT-2 is up-project, GELU, down-project. A gated FFN (Shazeer, 2020) adds a second up-projection and multiplies it, element by element, with the activated first one:

python
def swiglu(h, sd, pre="mlp."):
    return (F.silu(h @ sd[pre + "gate_proj.weight"].T) * (h @ sd[pre + "up_proj.weight"].T)) @ sd[pre + "down_proj.weight"].T

What the paper reports. With parameters and compute matched (the inner width shrinks to 2/3, from 3,072 to 2,048, to pay for the third matrix), held-out log-perplexity on T5-style span filling after 524,288 steps was 1.677 with ReLU, 1.679 with GELU, 1.636 with SwiGLU and 1.633 with GEGLU (lower is better). The author's explanation in full: "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence." Qwen3 uses SwiGLU, Gemma uses the GELU version (GeGLU, in sandwich_block above).

11. Mixture of experts: a router picks a few FFNs

Left, one FFN. Right, a mixture-of-experts FFN as in Mixtral and Jamba: a router scores every expert, the top two experts run and the rest are skipped, and their outputs are added with the router's weights.

A mixture-of-experts (MoE) layer replaces the single FFN with many FFNs, the experts, and a small router that picks a few for each token. Shazeer et al. (2017) put it between LSTM layers; Switch Transformers (Fedus et al., 2021) put it in place of the Transformer FFN with one expert per token and reported "up to 7x increases in pre-training speed with the same computational resources". Mixtral (Jiang et al., 2024) uses 8 experts and picks 2: 47B parameters in total, 13B used per token.

python
def moe_ffn(h, sd, top_k):
    p = (h @ sd["feed_forward.router.weight"].T).softmax(-1)        # (tokens, experts)
    w, idx = p.topk(top_k, -1)                                       # Jamba does not renormalize w
    out = torch.zeros_like(h)
    gu, down = sd["feed_forward.experts.gate_up_proj"], sd["feed_forward.experts.down_proj"]
    for t in range(h.shape[0]):
        for j in range(top_k):
            e = idx[t, j]
            g, u = (h[t] @ gu[e].T).chunk(2, -1)
            out[t] += w[t, j] * ((F.silu(g) * u) @ down[e].T)
    return out, idx

Since our function returns which experts it picked, we can look at the routing directly. In Jamba-tiny-dev, 8 experts with 2 per token, layer 3 sends ten of the 19 tokens to expert 0 and only one to expert 4. Layer 5 spreads the same tokens differently, with expert 3 the busiest at nine.

Which 2 of 8 experts each token of the sentence used in two MoE layers of Jamba-tiny-dev. Rows are the 19 tokens, columns the experts, filled cells the chosen ones. Layer 3 picks per expert: 10, 3, 7, 6, 1, 6, 3, 2. Layer 5: 2, 3, 5, 9, 5, 3, 6, 5.

This is one sentence in a small development model, so it shows how routing looks, not which expert does what. Cost stated by Mixtral: memory follows the total parameter count (47B), not the active one, and the routing adds overhead; they call MoE "more suitable for batched workloads". Switch lists "complexity, communication costs and training instability".

Changing the mixing cell

12. Mamba: one cell instead of attention plus FFN

Left, a Transformer layer with attention and FFN branches. Right, a Mamba layer: norm, project to x and a gate z, a short convolution over x, a selective scan with a fixed-size state, multiply by SiLU of z, project down. There is no separate FFN.

Attention looks back at every earlier token, so the cache grows with the input. Mamba (Gu and Dao, 2023) replaces it with a state of fixed size that is updated once per token. The update depends on the token (that is the "selective" part): a step size dt and the vectors B and C are computed from the input, so the model can decide per token how much to keep and what to write. The original Mamba also drops the FFN: the authors merged the two into one block "stacked homogenously".

This is the sequential version of the scan, from Jamba's Mamba layers:

python
def mamba1(h, sd, eps):
    T = h.shape[0]
    p = "mamba."
    xs, z = (h @ sd[p + "in_proj.weight"].T).chunk(2, -1)
    w = sd[p + "conv1d.weight"]
    xs = F.conv1d(xs.T[None], w, sd[p + "conv1d.bias"], padding=w.shape[-1] - 1, groups=w.shape[0])[0, :, :T].T
    xs = F.silu(xs)
    N, R = sd[p + "A_log"].shape[1], sd[p + "dt_proj.weight"].shape[1]
    dt, B, C = (xs @ sd[p + "x_proj.weight"].T).split([R, N, N], -1)
    dt, B, C = rms_norm(dt, sd[p + "dt_layernorm.weight"], eps), rms_norm(B, sd[p + "b_layernorm.weight"], eps), rms_norm(C, sd[p + "c_layernorm.weight"], eps)
    dt = F.softplus(dt @ sd[p + "dt_proj.weight"].T + sd[p + "dt_proj.bias"])   # step size per token
    A = -torch.exp(sd[p + "A_log"])                                  # (channels, N)
    state = torch.zeros_like(A)
    ys = []
    for t in range(T):
        state = torch.exp(dt[t][:, None] * A) * state + (dt[t] * xs[t])[:, None] * B[t][None]
        ys.append(state @ C[t] + sd[p + "D"] * xs[t])
    return (torch.stack(ys) * F.silu(z)) @ sd[p + "out_proj.weight"].T

The loop line is the whole idea: shrink the old state by exp(dt * A), write the new input through B, read out through C. (The three rms_norm calls on dt, B and C are Jamba's addition, not part of the original Mamba. Jamba also puts an FFN or MoE cell after each Mamba cell, as jamba_block in the repro bundle does, so its layers are not the FFN-free blocks of the Mamba paper.)

What the paper reports: 4-5 times the inference throughput of a Transformer of similar size, "since without the KV cache it can use much higher batch sizes", and Mamba-3B matching Transformers twice its size. Limitation stated: the evaluation is limited to small models. Jamba's paper adds one: a pure Mamba model "often does not follow the correct format" in few-shot prompts, which they read as weaker in-context learning.

13. Mamba-2: the same scan, simpler and in heads

Left, Mamba: project x and z, then convolution, then dt, B and C computed from x, then a scan with an A value per channel and state. Right, Mamba-2: x, z, B, C and dt all come from one input projection at the start, the scan has one scalar A per head, and a norm is added before the output projection.

Mamba-2 (Dao and Gu, 2024) makes three changes to the block. B, C and dt come straight from the input projection instead of being computed from x after the convolution. A becomes one number per head instead of a matrix per channel, which lets the scan be computed with matrix multiplications. A norm is added before the output, because "instabilities were prone to arising in larger models". The paper reports the new scan as 2-8 times faster than Mamba's, while allowing a state eight times larger.

Granite-4.0-H uses Mamba-2. Its layers, unlike pure Mamba, keep an FFN after the mixing cell, and scale both additions by 0.246. We verified Granite with the Mamba-2 function from the previous post, which also walks through it:

python
h = rms_norm(x, sd[p + "input_layernorm.weight"], eps)
mix, _ = mamba2(sd, p, cfg, h)       # in the 4 attention layers: attention with no rope call
x2 = x + mix * cfg.residual_multiplier                                # 0.246
h2 = rms_norm(x2, sd[p + "post_attention_layernorm.weight"], eps)
g, u = (h2 @ sd[p + "shared_mlp.input_linear.weight"].T).chunk(2, -1)
out = x2 + (F.silu(g) * u) @ sd[p + "shared_mlp.output_linear.weight"].T * cfg.residual_multiplier

Stacking

14. Hybrids: mostly Mamba, a few attention layers

The Mamba-2 paper itself tried mixing: at 350M parameters and 48 layers, perplexity on the Pile was 8.60 with no attention layers, 8.26 with six, and 8.68 for a Transformer (Table 2, lower is better), and the authors write that "around 10% of the total number of layers being attention performs best". Jamba compared one attention layer in four with one in eight and found "virtually no performance difference", so they chose one in eight.

Each model's config lists which layer is which, so we drew them straight from the configs:

One cell per layer for seven models, first layer on the left. GPT-2: 12 attention layers. Qwen3-0.6B: 28 attention. Gemma-3-1B: 26 layers, every sixth full attention and the rest sliding-window attention over 512 tokens. Jamba-v0.1: 32 layers, attention at 4, 12, 20 and 28, the rest Mamba, MoE on every other layer. Granite-4.0-H-350M: 32 layers, attention at 10, 13, 17 and 27. Granite-4.0-H-Small: 40 layers, attention at 5, 15, 25 and 35, MoE on every layer. Qwen3-Next-80B-A3B: 48 layers, every fourth layer full attention and the rest Gated DeltaNet, MoE on every layer.

The patterns differ more than the "hybrid" label suggests. Jamba repeats one attention layer every eight. IBM describes Granite-4.0-H Micro, Tiny and Small as Mamba-2 and attention "in a 9:1 ratio"; the small 350M model we ran has 28 Mamba-2 layers and 4 attention layers, and the attention layers sit at uneven positions. Qwen3-Next (model card) does not use Mamba at all: three of every four layers are Gated DeltaNet, a form of linear attention, and every fourth is full attention, with 512 experts per layer and 10 used per token. Gemma 3 is not a hybrid in this sense, but it stacks the same idea within attention: five sliding-window layers that only see the last 512 tokens for every full one.

What it buys. The cache only grows in the full-attention layers. We computed the cache for one sequence at 16 bits from each config (our script adds 2 × KV heads × head size × 2 bytes per token for each full-attention layer, the same capped at 512 tokens for sliding layers, and a fixed state for each Mamba or DeltaNet layer):

ModelLayers that grow with the inputPer tokenFixedAt 4K tokensAt 128K tokens
Qwen3-0.6B-Base28 of 28112 KiB0448 MiB14,336 MiB
Gemma-3-1B4 of 26 (22 capped at 512)4 KiB027 MiB523 MiB
Granite-4.0-H-350M4 of 324 KiB10.8 MiB27 MiB523 MiB
Jamba-v0.14 of 3216 KiB8.3 MiB72 MiB2,056 MiB
Qwen3-Next-80B-A3B12 of 4824 KiB37.7 MiB134 MiB3,110 MiB

These are calculated from each model's config at 16 bits, and the models are of very different sizes, so compare each row with its own attention count rather than with each other. One cross-check: our Jamba-v0.1 number at 256K tokens is 4,104 MiB, and the Jamba paper's Table 1 lists 4GB.

What it costs. A few attention layers still have to stay, with a cache that grows. In the previous post we switched off Granite's four attention layers. On an 18-token prompt, where Alice is 13 tokens back, the model still named her (0.988). When we stretched the same question to 1,410 tokens with unrelated paragraphs, her probability fell to 0.002, while the full model kept 0.990. Our reading is that the few attention layers carry what a fixed-size state loses. Choosing how many attention layers to keep, and where, is the design question these models answer differently.

15. Hybrid heads in one layer: Hymba

Left, a stacked hybrid as in Jamba and Granite: each layer has one kind of mixing cell, Mamba or attention, then an FFN. Right, a Hymba layer: attention heads and SSM heads read the same input side by side; each output is normalized and scaled by β, the two are added, then an FFN.

Hymba (Dong et al., 2024, NVIDIA) puts attention heads and state-space heads in the same layer, side by side, reading the same input. Each output is normalized and scaled before they are combined, because the SSM outputs were "consistently larger than those of the attention heads". At 300M parameters trained on 100B tokens, their Table 1 compares this parallel design with a sequential stack of the same parts: commonsense accuracy 45.19% vs 44.07%, recall 49.90% vs 45.16%. Hymba also uses sliding-window attention in most layers, keeping full attention in three.

The fifteen changes on one page

#ChangeWhat it replacedReported benefitCost or limit
1Post-LN block(2017 original)28.4 BLEU En-Deneeds warm-up (Xiong 2020)
2Decoder onlyencoder + decoderone stack for text generation (our summary)no source to cross-attend (our summary)
3Pre-LNPost-LNno warm-up, 40% faster BERT pre-trainingnone stated; stream grows with depth (ours, measured)
4RMSNormLayerNorm6.9% faster per stepnone stated
5Norm before and afterPre-LN"to stabilize training"no number given
6QK-normbias or soft-cappingstops logit blow-up at 8B (ViT-22B)no number for LLMs
7RoPElearned position rows27.5 vs 27.3 BLEUwhy it works not explained
8NoPERoPEbetter length generalization at 107Mnot tested at scale
9Parallel blocksequential blockabout 15% faster trainingsmall loss at 8B
10SwiGLUReLU/GELU FFN1.636 vs 1.677 log-perplexitynone stated
11MoE FFNone FFNup to 7x faster pre-trainingmemory holds all experts
12Mambaattention + FFN4-5x inference throughputweaker in-context format following
13Mamba-2Mamba scan2-8x faster scanrecall still limited by state
14Hybrid stacksall attention or all Mamba8.26 vs 8.60 perplexity at 350Msome layers keep a growing cache (ours, calculated)
15Hybrid headsstacked hybrid45.19 vs 44.07 commonsense at 300Mnone stated; more parts per layer (our reading)

Each benefit is the paper's own number, in its own setup, and so is each cost not marked "ours". What we ran is the code: all of it is in the repro bundle, and each block function reproduces its model's layer to rounding error.

Change one cell yourself

verify.py in the repro bundle below downloads the seven models from Hugging Face, records what goes into and comes out of each layer, and compares that with the block functions in blocks.py. It runs on a CPU. The easiest start is to change one line in a block function. In GPT-2's pre_ln_block, move the norm to after the addition and you have a Post-LN block; the gap from the real layer's output shows up at once. In the same way you can take RoPE out of qwen3_block.

If you want new posts in your inbox, the newsletter form is below.

Files for this post

Reproduction package

The harness, raw logs and result CSVs behind this post. No account needed; the README has a command that recomputes the tables without a GPU.

block-variants-repro.zip · 33 KB

Download

Stay Updated

Follow us for the latest posts and tutorials

Subscribe to Newsletter