The paper stopped at half the experts. A quarter loses four points instead of thirty, and a model trained without renormalization tells you why.

Follow-up to the previous post's reproduction of Xing Chen and Hengshuai Yao, arXiv:2609.04575. Same patch, same A100, same seed-0 MMLU subset of 2,000 items and GSM8K subset of 50. All numbers below are ours unless marked as the authors'.

In the previous post we confirmed the paper's headline: on Qwen3.6-35B-A3B you can activate 4 of the trained 8 experts, divide by the top-16 probability mass instead of the top-4, and MMLU moves by 0.30 points, which a paired test cannot distinguish from zero. The paper evaluates exactly that halving and stops. Two questions follow immediately, and neither is in the paper.

First: if the trick works at half, does it work at a quarter? Second: the paper's explanation is that training with renormalization calibrates expert gain to the training k. If that is the mechanism, then a model trained without renormalization should not need the trick, and should in fact be hurt by it. There is such a model. OLMoE-1B-7B ships with norm_topk_prob: false; its router uses raw softmax probabilities of the top 8 of 64 experts, never rescaled. That makes it a falsification test the authors could not run with two Qwen checkpoints.

Quarter and three-eighths on Qwen3.6-35B-A3B

Three experts with standard renormalization loses 12 points. Three experts with k₂=16 loses 1.2, and the paired test does not reach significance (p=0.13; 127 items flipped to wrong, 103 flipped to right). Two experts with standard renormalization is a collapse to 53. Two experts with k₂=16 is 78.5, a real loss of 3.95 points (p=6e-6) but a working model. GSM8K at n=50 does not separate 2:16 or 3:16 from the baseline at all; the 2:2 and 3:3 cells do collapse (24 and 62).

The curve the paper did not draw looks like this. With the denominator held at 16, going from 8 experts to 3 costs about a point, and to 2 costs about four. Without the trick, 3 costs twelve and 2 costs thirty. Whatever "the strength of the expert branch" means mechanically, most of the damage from cutting experts is in the gain, not in the missing experts, all the way down to a quarter of the trained k.

And the speed, in vLLM on the same A100 (decode tok/s at batch 1 / 8 / 32; gain against stock vLLM at 148.6 / 523 / 1,387, with the gain against the patched k=8 baseline in parentheses): k₁=3 gives 149.9 / 785 / 2,189, or 1.01× / 1.50× / 1.58× (1.12× / 1.61× / 1.63×); k₁=2 gives 153.2 / 847 / 2,131, or 1.03× / 1.62× / 1.54× (1.14× / 1.74× / 1.59×). At batch 1, dropping three quarters of the routed FLOPs is worth three percent against the engine you run; at batch 8 the curve keeps paying out to k₁=2. At batch 32 two experts comes out slower than three, which is the size of the generated-text confound from the previous post; that caveat applies at batch 8 too.

We did not tune k₂ for k₁=2 or 3. Sixteen was the authors' best value at k₁=4 and we used it unchanged. There may be a better one.

The model that was never renormalized

OLMoE-1B-7B-0125-Instruct, 64 experts, trained with top-8 and no renormalization. In our notation its native router is 8:64. We ran the same nine-cell grid: native k₂=E, forced renormalization k₂=k₁, and the paper's k₂=16, at k₁ = 8, 4, 2.

Renormalizing a model that was never renormalized destroys it. At the native k=8, simply dividing the top-8 probabilities by their sum, which is what the Qwen3-family and Mixtral routers do by default, takes OLMoE from 55.0 to 24.6 on a four-choice test, which is chance. GSM8K goes to zero. Nothing about the experts changed; only the scale of their weighted sum did. This is the mirror image of the Qwen result from the previous post, where removing renormalization cost 30 points. Each model is calibrated to whatever its training did.

The k₂ trick does not transfer. For OLMoE, at every k₁, the best denominator is the widest one, its native k₂=E. Narrowing to 16 is worse than native by 17 points at k₁=8, by 11.5 at k₁=4, and by 5 at k₁=2. There is no gain miscalibration to undo, so a narrower reference mass only inflates the branch.

Without the trick, cutting experts on OLMoE costs what you'd naively expect. Half the experts, native weighting: −5.35. A quarter: −16.85. Those are much steeper than Qwen with k₂=16 (−0.30 and −3.95) and much gentler than Qwen with standard renormalization (−6.15 and −29.45). OLMoE at reduced k is what an MoE looks like when the gain is right and the experts are simply missing.

What this says about the paper

The authors are careful to say the result may be specific to fine-grained MoE and to the Qwen series. The OLMoE grid says something sharper: the trick is specific to models trained with top-k renormalization, and its size is the size of the gain miscalibration that renormalized training leaves behind. If your model's config says norm_topk_prob: false, k₂ has nothing to fix. If it says true, as it does for Qwen3-MoE and DeepSeek-V3-style routers (Qwen3.5/3.6 renormalize unconditionally in code; Qwen1.5-MoE, by contrast, ships false), the paper's one integer is worth trying, and our numbers say it is worth trying below half.

Where we'd be wrong

We tested one non-renormalized model. If a second one (DeepSeek-V2-Lite also ships with norm_topk_prob: false, softmax scoring, 64 experts, top-6) shows a k₂ benefit at reduced k₁, the "nothing to fix" reading is wrong and we will say so. We also did not test k₁=1 on Qwen, or any k₂ other than 16 below k₁=4; a k₂ sweep at k₁=2 might close some of the 3.95-point gap or widen it.

Weaknesses

Everything from the previous post applies: seed-0 subset, GSM8K n=50 with our prompt format, no long-form generation. OLMoE is a 1B-active model and its MMLU baseline of 55 leaves less room to fall before the chance floor, so absolute drops are not comparable across the two models; the comparison that holds is within-model, native versus modified. OLMoE speed was not measured in vLLM; the HF loop gives 44–50 tok/s at every cell, which, as the previous post shows, says nothing.

Scripts and raw results: drafts/moe-k2-exp.py, drafts/moe-k2-results.jsonl, drafts/moe-k2-olmoe-results.jsonl. The same self-test verifies that the patched OLMoE router at 8:64 is bit-identical to the stock one.

Next in the inbox: the newsletter form is below.