896 Experts, None Dead: What MoE Load Actually Looks Like at Token Level

There’s a seductive idea floating around MoE deployment discussions: with huge scale-up domains coming (NVL72 and beyond), why not give every expert its own accelerator? One expert, one chip, perfect disaggregation.

We measured what that would actually do, using Kimi K3 — 896 routed experts per MoE layer, 92 such layers, top-16 routing — served on an 8×B300 node. A small patch in the serving stack recorded a per-layer histogram of every expert selection across a mixed workload (math, knowledge, and code benchmarks). Here’s what the router’s traffic really looks like.

Load is very uneven…

If routing were uniform, each expert would see 1/896 ≈ 0.11% of tokens. Reality:

• The hottest expert per layer runs ~22× uniform load on a mixed workload, averaged over layers. The worst layer’s hottest expert: 35×.
• The standard deviation of per-expert load is about 2× the mean (CV ≈ 2, per-layer range 0.7–3.9).

So expert traffic has a serious heavy tail. Any deployment that assigns equal capacity to every expert is provisioned for the average and bottlenecked by the maximum.

…but nothing is dead

• Zero dead experts. Across every task we ran, every one of the 896 experts in every layer got traffic.
• The coldest experts still ran at roughly half of uniform load, pooled across tasks.

That’s a healthy router: no wasted parameters, no collapsed experts — just an uneven distribution over experts that are all pulling weight.

The hot set moves with the workload

Here’s the part that matters most for deployment. We correlated per-expert load across task types:

• ARC ↔ MMLU (both knowledge-flavored): Pearson r = 0.69. Related workloads, similar hot sets.
• GSM8K ↔ HumanEval (math vs code): r = 0.27.
• ARC ↔ HumanEval: r = 0.10 — essentially uncorrelated.
• The top-5% hottest experts overlap between tasks by only 11–40%.

The hot set isn’t a property of the model. It’s a property of the model and the traffic mix — and the traffic mix changes by the hour on any real serving cluster.

What one-accelerator-per-expert would cost you

Back to the seductive idea. If the hottest expert per layer carries 22× uniform load, a no-replication layout runs the whole domain at roughly 1/22 of its hot-expert-limited throughput. Ninety-five percent of your silicon idles while one chip melts.

The fix is old and cheap: replicate hot experts. We simulated proportional replication (EPLB-style — extra slots assigned to experts by measured load):

Ten percent more expert slots cuts the straggler from 22× to 3.6×. Twenty-five percent gets you to 2.6×. After ~1.5× you’re deep into diminishing returns — the residual ~2× tail comes from within-layer heavy-tailedness that proportional replication can’t fully flatten (in practice, systems cap it with capacity factors or route-around).

The catch: static layouts won’t hold

Given the weak cross-task correlation above, a replication layout tuned on Monday’s traffic is mistuned for Tuesday’s. A code-heavy burst reshuffles the hot set that a knowledge-heavy mix established. Two implications:

1. Dynamic rebalancing wins. DeepSeek-style EPLB with periodic re-placement — measure load, re-replicate, repeat — fits this data far better than any baked-in layout.
2. Per-expert disaggregation without replication is the worst point in the design space: maximum hardware, minimum throughput, and the imbalance moves around underneath you.

None of this is exotic. It’s a bincount patch, a weekend of analysis, and one 8.7-hour session on a rented 8×B300 node. If you’re planning MoE serving topology for models of this class, measure your router’s traffic before you buy the org chart — the data is cheap and it will probably surprise you.