The Free Lunch That Wasn't (Part 2: Ten Seeds Later)
This is part 2 of two. Part 1 set up the experiment and ended on a +5.5 point HumanEval gain from routing to 32 experts instead of 16, flagged as single-seed and unbelieved. This post is the replication.
To recap the premise: Kimi K3 is a 1.56 TB, 896-expert mixture-of-experts model.
For every token, at every one of its 61 MoE layers, a router picks 16 experts
out of 896 and blends their outputs. That top-k = 16 is a training-time
hyperparameter, and you can override it at inference time with a ~10-line
patch to the serving stack. No retraining, no fine-tuning.
Part 1 asked whether K3’s top-16 routing is well-calibrated or leaving quality on the table. It came back with one tantalizing number — HumanEval 81.7% at top-32 versus 76.2% at baseline — and an explicit refusal to believe it on a single seed over 164 problems.
Two things had to happen before that number meant anything.
First: look inside the router
Before re-rolling dice, we instrumented the router itself. A patch in vLLM’s expert-selection path recorded, for every token at every MoE layer, two numbers over the full 896-expert sigmoid distribution: how much of the total routing mass the selected top-16 actually captured, and how many experts you’d need to cover 90% of the mass (k90). Across ~420 million router calls spanning math, knowledge, and code prompts:
| task | k90 (median) | top-16 mass (median) |
|---|---|---|
| gsm8k | 698 | 5% |
| arc | 684 | 6% |
| mmlu | 687 | 5% |
| humaneval | 697 | 5% |
Read that again: the 16 experts K3 actually uses capture ~5% of the router’s probability mass, and covering 90% of the mass takes ~700 of 896 experts. The distribution is almost flat — the sigmoid router operates on tiny relative score differences, not concentrated mass. And it’s task-independent: math, code, and knowledge prompts produce essentially the same profile.
This kills an entire family of ideas by construction. “Adaptive-width” routing — pick experts until you’ve covered τ of the mass, like top-p sampling — degenerates to uniform maximum-width routing for any reasonable τ, because no small expert set ever accumulates meaningful mass. We had a top-p routing sweep pre-registered, gated on finding task-dependent structure. The gate failed. The sweep was never run. (Pre-registering your gates is cheap insurance against running experiments that can’t mean anything.)
It also reframes part 1’s result before a single new eval runs. If the top-16 selection captures 5% of the mass, then “select 32 instead” is not sampling deeper into a concentrated distribution — it’s grabbing more of a flat one. There was never a strong mechanistic reason to expect a large win.
Then: the replication
The main event: top-16 vs top-32, 10 seeds per arm, on harder variants — HumanEval+ (hardened tests) and MBPP+ (378 problems, tighter error bars).
| benchmark | top-16 | top-32 | Δ | verdict |
|---|---|---|---|---|
| HumanEval+ | 63.8 ± 3.3 | 65.8 ± 1.9 | +2.0 | not significant (t = 1.67) |
| MBPP+ | 93.3 ± 0.7 | 91.6 ± 0.5 | −1.7 | significant regression (t = −5.74) |
The +5.5 evaporated. Seed-to-seed standard deviation on HumanEval+ alone is ±3.3 points — the round-1 result was mostly noise wearing a lab coat. And on the larger, tighter MBPP+, doubling the routed experts is a statistically significant regression, while also costing ~2× the MoE FLOPs per token.
Worth sitting with the shape of that: the headline benchmark from part 1 came back merely insignificant, but the benchmark with twice the problems and a quarter the variance came back significantly worse. If we had stopped at HumanEval+ and shrugged at “+2.0, not significant,” we’d have missed that wider routing actively costs you something.
What we actually learned
K3’s router is doing its job exactly as trained. Expert identity and expert weight mass are both calibrated; widen the selection and you dilute it, prune the mass and you break it (part 1’s RUN 2 showed the second half of that). The free lunch was never there — but now that’s a measured fact instead of a hunch.
The broader lesson costs nothing to repeat: a single-seed result on a 164-problem benchmark is not a result. A ±3-point seed swing is enough to manufacture a publishable-looking delta from thin air. Part 1 exists in its original form precisely because that’s what the honest intermediate state looked like — a promising number, correctly distrusted. The failure mode isn’t getting a noisy result; it’s publishing it as a finding.
If a number would change your roadmap, buy the ten seeds.
The whole investigation — three eval rounds on a model that needs 2+ TB of GPU memory to serve — took about 13 hours of 8×B300 node time on Verda, most of it in a single overnight run each round.