One integer halves MoE expert compute. We measured the speed the paper didn't: free at batch 8, not at batch 1.

Xing Chen and Hengshuai Yao, arXiv:2609.04575, submitted 2026-09-04. No code released. All author numbers below are from the paper's Table 5 and Introduction. All other numbers are ours, run on 2026-09-08.

Every fine-grained MoE you run does a small thing at every layer: it picks the top-k experts by router probability, then divides those k probabilities by their sum so they add to one. The paper's claim is that this renormalization is not neutral. During training it calibrates the gain of the expert branch to the training k. Drop k at inference and you change which experts fire and also how loud the whole branch is. The fix is one integer: activate the top-k₁ experts, but divide by the probability mass of the top-k₂. Eq. 2 in full: w_i = p_i / Σ_{j ∈ Top-k₂} p_j for i ∈ Top-k₁; standard renormalization is k₂ = k₁, and k₂ = E (all experts) is no renormalization. k₁ sets compute, k₂ sets gain. It adds no parameters and needs no training.

The paper has two holes we could fill in a day. There is no code, and there is no speed measurement of any kind. "Halving routed-expert compute" is a FLOP statement. So we wrote the patch, reproduced the accuracy table, and timed it.

What we ran

Qwen3.6-35B-A3B (256 experts, trained with k=8, plus one shared expert that the method leaves alone), bf16, on a single A100 80GB. The model sits at 65.4 GiB resident. The authors used a 98 GB accelerator; nothing about the method needs it.

The patch replaces one method on Qwen3_5MoeTopKRouter in HF transformers 5.16.1. It computes the top-k₂ probabilities, takes the first k₁ of them as the active set, and divides by the top-k₂ sum. With k₁=k₂=8 the output is bit-identical to the stock router (we check this in a self-test). With k₂=E (all 256) it returns raw softmax probabilities, which is the "no renormalization" case.

Accuracy: MMLU 5-shot on 2,000 test questions, drawn with a fixed seed from the 14,042-item test set (the paper also uses n=2,000 but does not say how its subset was drawn). GSM8K 0-shot on 50 questions, chat template with thinking off, greedy, answer parsed after ####. Both are scored per item so we can run the same paired exact McNemar test the authors use.

Speed: batch-1 decode in HF transformers, then in vLLM 0.28 with the same routing rule monkeypatched into FusedTopKRouter and the model built with num_experts_per_tok=k₁ so every kernel buffer is sized for the smaller k.

The accuracy table replicates

Authors' numbers are the Qwen3.6-35B-A3B rows of Table 5. Our subset is not theirs, so the absolute levels differ by about a point; the pattern is what matters.

Standard renormalization at half the experts costs 6.15 points on our subset (the authors report 4.65). Widening the denominator to 16 brings it back to 0.30 under the full model, and the paired test cannot tell the two apart (p=0.73; the authors report p=0.66). Removing renormalization entirely loses 30 points. The authors call that case catastrophic and report a 27.45-point drop at k₁=6.

GSM8K at n=50 is too small to claim anything beyond "no collapse at k₂=16 and a real collapse at k₂=256" (b=34 base-right-then-wrong, c=0, p=1.2e-10).

The speed the paper didn't measure

In HF transformers, halving the experts saves nothing. Same protocol as the vLLM runs below (random 512-token prompts, 128 new tokens, median of 5): decode ran at 14.9 tok/s with 8 experts and 15.4 with 4, and every condition in between landed in the same 14.5–15.4 band. The eager expert loop is not the bottleneck at batch 1; the Gated DeltaNet layers, the Python dispatch, and the memory traffic of everything that isn't an expert are. If you measure the method in transformers, you will conclude it does nothing for latency. That conclusion would be wrong, but only because of the runtime.

In vLLM the gain exists at batch 8, and it does not exist at batch 1. Decode throughput on one A100, random 512-token prompts, 128 new tokens, median of 7 rounds. Two baselines: stock vLLM, which is what you run today, and the same routing rule at the unchanged k=8 through our Python patch, which isolates the method from the patch's own cost.

Read the two gain columns together. Against the patched baseline, halving the experts is worth 1.10× at batch 1: that is the method with its implementation cost held out. Against the vLLM you actually run, it is worth 0.99×. Our routing runs in Python, not in vLLM's fused top-k kernel, and that costs 9.5% at batch 1, which is exactly what the halved experts buy back. Decode at batch 1 is bandwidth-bound, and expert weights are a minority of what this hybrid architecture reads per token, so there was never much to gain there. The shim's overhead shrinks with batch (9.5%, 7.0%, 3.2% at 1, 8, 32), so at batch 8 the method is worth 1.35× against stock and 1.45× against the patched baseline, and those two numbers are close because the expert matmuls now matter.

One more thing the table shows, which we did not expect. At batch 32, 4:4 and 4:16 differ by 17% (1,651 vs 1,938 tok/s) even though they run identical kernels. The top-k₁ selection does not depend on k₂ at all, only the weights do, so on identical input the routing is identical; prefill times confirm it (638 vs 639 ms). The decode difference can only come from the generated text diverging, and the fused MoE kernel's cost depends on how the batch's tokens spread across experts. That confound applies at batch 8 as well as 32, only smaller, so read both of those columns as ranges. The clean fix is to force identical token sequences through every condition; we have not done that yet.

One check on the vLLM numbers: k₂ only changes a denominator, so 4:4, 4:16 and 4:256 should be equally fast at batch 1, where text divergence barely matters. They are, within 1% (147.7 vs 149.0 tok/s); the k₂=256 cell is slower, 136.4, because the shim then sorts all 256 probabilities.

Where we expected to be surprised, and where we actually were

We wrote our guesses down before running anything. Batch-1 gain below 2×, somewhere in 1.2–1.7×: the measured gain is 1.10× against the patched baseline and 0.99× against stock vLLM, below our range either way. GSM8K degrading more than MMLU: not visible at n=50, where every k₂=16 cell matches the baseline. Quantization interacting with k₂: we dropped this hypothesis before running, because the model fits an 80 GB card in bf16 and we had no reason to add a confound. The real surprise was the transformers result. We had assumed the framework would show some of the FLOP saving, and it showed none.

What to do with it

If you serve Qwen3.6-35B-A3B in vLLM and are throughput-bound at batch, k₁=4 with k₂=16 is 1.35× at batch 8 against stock vLLM (1.45× against the patched baseline) for a change the paired test cannot distinguish from the full model on MMLU. At batch 1 it is worth nothing with a Python router, and about 10% with a fused one that does not exist yet. Two things we did not test: long-form generation (the authors list it as untested too) and any model outside the Qwen3.x family. The second one we ran the next day: on a model trained without renormalization the trick does nothing. That is the next post.

Weaknesses

Single fixed-seed subset of MMLU, not the authors' subset. GSM8K n=50, our prompt format, thinking disabled; the paper does not specify its GSM8K format. Speed was measured on random-token prompts of 512 tokens with 128 new tokens, seven rounds, median; real prompts have different KV traffic. The vLLM patch is a research shim, not a fused kernel; we report gains against both the shim and the stock engine so its cost is visible. One GPU, one model family, and the generated-text divergence between conditions at batch 8 and 32 is unresolved.

Patch, evaluation script, vLLM script, raw JSONL results and the chart script are in the repo under drafts/moe-k2-*. Seed 0 everywhere.

If you want the next one in your inbox, the form is below.