Paper of the Week #3 — Half the FLOPs Is Not Half the Time

Paper of the Week #3 — Half the FLOPs Is Not Half the Time

A new paper halves the expert compute of a fine-grained MoE with one integer and no training, and its accuracy table replicates on our A100 to within a point. What it never reports is time. We measured it: nothing in HF transformers, nothing at batch 1 in the vLLM you run today, 1.35× at batch 8.

Sep 3 – Sep 9, 2026 · Training-Free Halving of Activated Experts · Don't Drop Dropout · Repo-To-Skill · Scale-QLoRA

The Claim

A FLOP count is a claim about the model. The bill is written by the runtime.

Efficiency papers report the number they can compute from the architecture: FLOPs saved, parameters skipped, experts not activated. That number is real, and it is not the one you pay. What you pay is wall-clock on a specific runtime, where the saved operation may never have been on the critical path. Last issue's complaint was that harness papers publish a score without a denominator. This issue's is the mirror of it: efficiency papers publish a denominator without the score. This week's lead paper is the cleanest case I have seen, because everything it does report is correct, and the one number it leaves out changes what the result is worth by a factor that depends entirely on which serving stack you run.

The Receipt: the paper's table, then the column it doesn't have

Xing Chen and Hengshuai Yao (arXiv:2609.04575, submitted September 4, no code) make a small, precise claim. Fine-grained MoE routers pick the top-k experts and renormalize their probabilities to sum to one. Training under that renormalization calibrates the expert branch's gain to the training k, so cutting k at inference changes the branch's loudness as well as its membership. Their fix separates the two: activate the top-k₁ experts, divide by the mass of the top-k₂. Eq. 2 in full: w_i = p_i / Σ_{j ∈ Top-k₂} p_j for i ∈ Top-k₁, where standard renormalization is the special case k₂ = k₁ and k₂ = E (all experts) is no renormalization at all. Their headline, on Qwen3.6-35B-A3B: 4 of 8 experts costs 4.65 MMLU points with standard renormalization and 0.35 with k₂=16.

I wrote the patch (one method on the HF router, bit-identical to stock at k₁=k₂=8), loaded the model in bf16 on one A100 80GB, and scored MMLU 5-shot on a fixed-seed 2,000-question subset, per item, with the same paired McNemar test the authors use.

The table replicates. The k₂=16 recovery is 0.30 points on our subset against the authors' 0.35, and the test cannot separate it from the full model (their p=0.66, ours 0.73). Four rows the paper does not have: three experts with k₂=16 sits 1.2 points under the full model, still inside noise, where standard renormalization (3:3) loses 12.45; two experts with k₂=16 sits 3.95 under, a real loss and a working model, where standard renormalization (2:2) collapses to 53. So far the paper is right about everything it measured, and generous to itself about nothing.

Then the column it doesn't have. "Halving routed-expert compute" is a FLOP statement, and the paper reports no throughput, no latency, no hardware for inference. I measured batch-1 decode in two runtimes.

In HF transformers 5.16.1 (same protocol as below: random 512-token prompts, 128 new tokens, median of 5), halving the experts changes nothing: 14.9 tok/s at k=8, 15.4 at k₁=4, every condition inside a 14.5–15.4 band. The eager expert loop is not what the GPU is waiting on. If you evaluate this method where most researchers evaluate methods, you will conclude it does nothing for latency.

In vLLM 0.28, with the same rule monkeypatched into the fused router and the model built with num_experts_per_tok=k₁ so kernel buffers shrink with it:

Two baselines, and the gap between them is this issue's point. Against the patched 8:8 (same Python routing, k unchanged), 4:16 gains 1.10× at batch 1 and 1.45× at batch 8: the method with its implementation cost held out. Against stock vLLM, the engine you run today, batch 1 is 0.99×. The Python shim costs 9.5% there, and halving the experts buys back exactly that. Two of eight experts, three quarters of the routed FLOPs gone, is 1.03×. The shim's overhead falls with batch (9.5%, 7.0%, 3.2%), so the larger-batch gains survive it: 1.35× at batch 8 and 1.40× at batch 32 against stock.

Batch 1 is flat because decode there is bandwidth-bound and expert weights are a minority of what this hybrid Gated-DeltaNet model reads per token. Batches 8 and 32 are ranges, not points. k₂ changes weights, never which experts are selected, so routing on identical input is identical and prefill times agree (638 vs 639 ms); yet at batch 32, 4:4 and 4:16 differ by 17% on the same kernels (1,651 vs 1,938 tok/s). The generated text diverges, and the fused MoE kernel's cost depends on how a batch spreads across experts. The confound is smaller at batch 8, not absent, and at batch 32 two experts even comes out slower than three (1.54× vs 1.58×). Forcing identical token sequences through every condition is the fix; it is on the list for issue #4.

The honest summary of "halves expert compute" on this model: 1.0× in transformers; in vLLM, 1.10× at batch 1 against an implementation-cost-free baseline and 1.00× against the vLLM you run today; 1.35× at batch 8 against stock, 1.45× against the shim. The title says half the FLOPs is not half the time. The table says a quarter of the FLOPs is three percent of the time at batch 1. None of this is in the paper.

One more control, because the mechanism deserved a test the authors could not run with two Qwen checkpoints. OLMoE-1B-7B was trained without renormalization. If the paper's story is right, k₂ should have nothing to fix there. It doesn't: native OLMoE scores 55.0 on the same MMLU subset, forcing renormalization on it drops it to 24.6 (chance), and at every reduced k₁ the widest denominator is the best one. The asymmetry is itself evidence. The top-k mass is below one, so dividing by it amplifies the branch and dropping the division attenuates it; Qwen with its renormalization removed loses 30 points, OLMoE with renormalization added loses 30 points from a lower start and lands at chance. An amplification the model never learned to expect is the more destructive of the two, which is what a gain-calibration story predicts. The trick is real, and it is specifically a correction for renormalized training.

The Contrast

Don't Drop Dropout — the same claim, one step further from your GPU

Elhoushi et al. (Cerebras, arXiv:2609.05275) argue that layer dropout belongs back in LLM pretraining: with the right per-layer distribution and schedule, the same validation loss at up to 25% fewer training FLOPs, and models that allow early exit and self-speculative decoding "yielding up to 1.5× inference speedup with negligible accuracy loss". 2,400 runs from 271M to 8.2B, all pretraining on Cerebras CS-3 per the abstract, no code mentioned. I don't doubt the loss curves. But "1.5× inference speedup" is this issue's kind of number: a skipped layer is a FLOP saved, and whether it is a millisecond saved depends on whether that layer sat on your runtime's critical path. Reproducing it at 271M on a GPU would take days, not a day.

Repo-To-Skill — the denominator hidden the other way

Chen et al. (BAAI, arXiv:2609.02749) distill 1,000 ML repositories into 5,000-plus "skills" for a research agent. With backbone and budget fixed, it "scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench". Relative gains, and the abstract gives no absolute score for either arm; 134% over a baseline of 4 is a different result from 134% over 40. It is the FLOP count's move from the other side, a true number that does not tell you what you'd get. Read the absolute tables before repeating the headline.

Scale-QLoRA — the one I can't use yet

arXiv:2609.04526 claims that merging a LoRA into a native NVFP4/MXFP4 checkpoint the naive way re-derives the quantization codes and can erase the adaptation, up to 39 points, and that training only the block scales avoids it at a 3.9× per-step training speedup on 8B. The failure mode alone would be worth a post. But I have only the abstract: the HTML would not load this week, no code release I could confirm, and a speedup with no hardware. A reproduction candidate, not evidence.

Where I'd Be Wrong

Two results would overturn this issue's reading. First, a fused k₂ kernel: if a real kernel implementation delivers 1.5× or more at batch 1 on this model, then expert weights are not the minority of per-token traffic I claim, and the "bandwidth-bound" explanation is wrong. I will score that when one exists, mine or anyone's. Second, DeepSeek-V2-Lite also ships norm_topk_prob: false. If k₂ helps it at reduced k₁, the "nothing to fix without renormalized training" reading from the OLMoE control is wrong. That one I can run, and will, for issue #4.

Ship It · Code of the Week

The whole method is this, on any HF transformers 5.16 router class whose name ends in TopKRouter (Qwen3_5MoeTopKRouter, OlmoeTopKRouter). MIT; the full script is linked in the footer.

def forward(self, hidden_states):                     # k1 = experts run, k2 = reference mass
    k1, k2 = self.k1, self.k2                          # k2 >= k1; k2 == num_experts means no renorm
    hidden_states = hidden_states.reshape(-1, self.hidden_dim)
    logits = F.linear(hidden_states, self.weight)
    probs = torch.softmax(logits, dtype=torch.float, dim=-1)
    top_vals, top_idx = torch.topk(probs, min(k2, self.num_experts), dim=-1)
    scores = top_vals[:, :k1] if k2 >= self.num_experts else top_vals[:, :k1] / top_vals.sum(-1, keepdim=True)
    return logits, scores.to(logits.dtype), top_idx[:, :k1]

Qwen3_5MoeTopKRouter.forward = forward               # or OlmoeTopKRouter
for m in model.modules():                            # k1 = 4, k2 = 16 is the paper's setting for a trained k of 8
    if type(m).__name__.endswith("TopKRouter"):
        m.k1, m.k2 = 4, 16

Monday morning, two things. If you serve a renormalized fine-grained MoE and are throughput-bound, try k₁ at half the trained k with k₂ at twice it; on Qwen3.6-35B-A3B that is 1.35× at batch 8 against stock vLLM for a change the paired test cannot see, and nothing at batch 1. And whichever efficiency method you evaluate next, measure tokens per second in the runtime you serve, not the one you research in. That includes mine: the numbers above come from a Python patch, not a kernel, and I report them against the unpatched engine for exactly that reason. The two runtimes disagreed by the entire effect this week.

The Ledger

Issue #2 promised two things. The iso-cost control ran: the single agent given the wrap's budget as three independent attempts on the same eight hidden-test tasks. Result: pass@3 8 of 8, the same as the wrap's 8 of 8, at $6.53 against the wrap's $5.98 (1.09×) and 191,584 against 177,100 in+out tokens (1.08×). Per attempt the single agent passed 22 of 24 (92%); issue #2's single run was 7 of 8. The one task the wrap "won" last issue, rest-api, passed on 2 of 3 single attempts this time, so it was a coin-flip miss, not a capability gap. Two caveats: pass@3 is the upper bound of best-of-3 (a real selector would have to pick without the hidden tests), and these are the same eight katas, not the repo-level tasks I promised. Kept on cost, not on task scale: counted as partial. The second promise, HoH at ten iterations under a matched budget, did not run: their artifacts still do not permit a matched-budget read, and I am not going to fund a ten-iteration harness sweep to grade someone else's curve. Not kept, counted as such; it stays on the ledger until the artifacts change.

#2's claim, that the wrap's gain sits inside rerun noise at three times the tokens, stands and is sharper now: at equal spend, three plain attempts match the wrap's pass rate exactly, so the wrap bought nothing that reruns could not.

Reproduction streak: 3 weeks. Next issue: DeepSeek-V2-Lite as the second non-renormalized control, and Scale-QLoRA's naive-merge failure if the body and code surface.

Patch and evaluation: moe-k2-exp.py (HF, self-tested bit-identical at stock k), moe-k2-vllm.py (vLLM 0.28, one process per condition), raw JSONL for every cell. Qwen3.6-35B-A3B bf16 on one A100 80GB, 65.4 GiB resident. MMLU: 2,000 items, seed 0. Speed: random 512-token prompts, 128 new tokens, median of 7 rounds in vLLM and 5 in transformers, machine otherwise idle. Limits: one GPU, one model family for the speed numbers, a Python routing shim rather than a kernel, and generated-text divergence between conditions at batch 8 and 32. The two measured posts behind this issue: the reproduction and speed, and the quarter and the OLMoE control.

Iso-cost control: potw3-isocost.py, 8 tasks × 3 single calls, sonnet via Claude Code 2.1.266, graded by the runner on the hidden official pytest suites, $6.53 total.

Wall-clock: ~2 h model download, ~6 h GPU for the accuracy sweeps, ~30 min vLLM, ~35 min iso-cost. Verified on: 2026-09-08 (MoE), 2026-09-09 (iso-cost).

References

• Chen and Yao, Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models
• Elhoushi et al. (Cerebras), Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
• Chen et al. (BAAI), Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills
• Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs
• OLMoE-1B-7B-0125-Instruct, the non-renormalized control
• Paper of the Week #2 — The Score Is in the Abstract, the Bill Is in Table 2