Hybrid Mamba-Transformer, Measured: Qwen3.5-9B's Cache Is 4.4x Smaller Than Qwen3-8B's — and Why Our Speed Numbers Don't Count

Hybrid Mamba-Transformer, Measured: Qwen3.5-9B's Cache Is 4.4x Smaller Than Qwen3-8B's — and Why Our Speed Numbers Don't Count

Our first post on hybrid Mamba-Transformer models argued from three papers that the 2026 convergence — three quarters linear layers, one quarter attention — buys a much smaller inference cache and faster long-context decoding. It cited "~25% of full KV cache" and "~2–3x inference speed" from the papers. This post checks the first claim on hardware we control and explains, carefully, why we can only *partly* check the second.

Setup: Qwen3.5-9B (32 layers: 24 Gated DeltaNet + 8 full attention, full_attention_interval=4) against Qwen3-8B (36 layers, all attention), both bf16, one A100 80GB, batch 1, contexts from 2K to 64K tokens, HuggingFace Transformers 5.16 with the flash-linear-attention Triton kernels. We measure the cache — key/value tensors for attention layers, recurrent and convolution state for linear layers — by summing every tensor in the cache object, not by reading GPU memory (which would include logits).

1. The Cache: 4.4x Smaller at 64K, Approaching 4.6x

The numbers decompose exactly, which is the satisfying part.

Transformer: 36 layers × 8 KV heads × 128 head-dim × (K+V) × 2 bytes = 147 KB per token. 61,839 tokens → 8.49 GB. ✓

Hybrid: only the 8 attention layers grow with context. They use 4 KV heads × 256 head-dim × 2 × 2 bytes = 32 KB per token. The 24 Gated DeltaNet layers hold a *fixed-size* state regardless of context — a 32-head × 128 × 128 recurrent matrix plus a 4-wide convolution buffer per layer, about 25 MB total in bf16. So cache ≈ 25 MB + 32 KB × tokens. At 61,839 tokens: 1.94 GB. ✓

Two things follow. The asymptotic ratio is 147 / 32 = 4.6x, not the "4x" you'd get from the 8/32 layer count, because Qwen3.5 also uses fewer KV heads. And at short contexts the ratio is lower (2.4x at 2K) because the fixed linear state is a larger share — hybrids pay a small constant cost up front and win increasingly as context grows. The first post's "~25% of full KV cache" was, if anything, conservative.

For serving, this is the number that matters: at 64K context, one A100 holds roughly 30 concurrent Qwen3.5-9B sequences in the memory it takes to hold 7 Qwen3-8B sequences. That is the hybrid's actual product.

2. Speed: What We Measured, and Why It Doesn't Answer the Question

Here is the honest part. We also timed prefill and decode in HuggingFace eager mode:

Read naively, the hybrid *loses* on prefill and ties on decode. That reading is wrong, and it is worth understanding exactly why, because it is the same trap anyone benchmarking a new architecture in a research framework falls into:

• Decode at ~25 tok/s for both is the Python loop, not the model. An A100 decodes an 8B model at 130–150 tok/s in llama.cpp or vLLM; HuggingFace eager decode is bound by per-step framework overhead, which is identical for both architectures. The measurement has no resolution left to see a KV-bandwidth difference.
• Hybrid prefill is kernel-bound in a way the transformer isn't. The transformer's attention runs on PyTorch's fused SDPA kernels, tuned for years. The Gated DeltaNet layers run on flash-linear-attention's Triton kernels — and we could not build causal-conv1d against this machine's CUDA 12.1 toolchain, so the convolution falls back to a slow path. The first 2K-context run took 42 seconds, almost all of it Triton compilation. The hybrid's prefill throughput *doubles* from 8K to 64K (746 → 3,519 tok/s) while the transformer's *falls* (9,877 → 5,400), which is exactly the linear-vs-quadratic shape the theory predicts — but at 64K the hybrid is still at 65% of the transformer, because of kernel maturity, not architecture.

So: this experiment confirms the memory claim and cannot confirm or refute the speed claim. A fair speed comparison needs a serving engine with production kernels for both paths — vLLM added Qwen3.5 hybrid support (and even TurboQuant for its attention layers) this spring, and that is the follow-up we are running next.

A note on peak memory, since it surprised us: peak allocation during 64K prefill was *higher* for the hybrid (47.7 GB vs 41.7 GB). That is not the architecture either. Qwen3.5's vocabulary is 248K tokens versus Qwen3's 152K, and HuggingFace materializes logits for every position during prefill: 61,839 × 248,320 × 2 bytes ≈ 30 GB. A serving engine never does that.

3. What Changed From the First Post

One correction to our own framing: the first post described the saving as coming from "75% linear layers." The larger lever in Qwen3.5-9B is that its 8 attention layers use 4 KV heads where Qwen3-8B uses 8. Half of the per-layer saving is GQA width, not the hybrid split. Architecture comparisons across model families always bundle several decisions; the cache arithmetic above is how to un-bundle them.

4. Reproduce

python -m venv ~/venvs/hybrid && ~/venvs/hybrid/bin/pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128
~/venvs/hybrid/bin/pip install "transformers>=5.9" flash-linear-attention
CUDA_VISIBLE_DEVICES=0 python bench-hybrid-hf.py --model Qwen/Qwen3-8B   --out hybrid-hf.json
CUDA_VISIBLE_DEVICES=0 python bench-hybrid-hf.py --model Qwen/Qwen3.5-9B --out hybrid-hf.json

The script (attached) sums numel × element_size over every tensor reachable from past_key_values, which is the only way to get an apples-to-apples cache size when one model's cache is a DynamicCache of K/V tensors and the other's carries recurrent states.

References

• Part 1: Hybrid Mamba-Transformer MoE — Three Teams, One Architecture
• Qwen3.5 model card and config (layer_types, full_attention_interval) on Hugging Face
• flash-linear-attention — Triton kernels for Gated DeltaNet
• vLLM PR #39931 — hybrid-model support used in the follow-up