Does Kimi K3 Want More Experts? (Part 1: The Tempting Result)

This is part 1 of two. It ends with a result that looks like a free lunch. Part 2 spends ten seeds and a router instrumentation pass finding out whether it was real. It wasn’t.

Kimi K3 activates 16 of its 896 experts per token. That number comes from training; nothing forces you to keep it at inference time. The router scores experts, and how many of those scores you act on is a runtime choice.

Two questions, one afternoon on eight B300s:

1. Does the model do anything useful with more experts? Top-32 instead of top-16, at roughly double the routed FLOPs.
2. Can you get better picks for free? Select the top 32, normalize weights across all 32, then keep only the best 16 — the same expert budget, chosen against a wider field. A distribution over 32 candidates carries different information than one over 16, so it might identify a better 16.

How K3 routes

Sigmoid scoring across 896 experts, grouped top-k selection (noaux_tc), keep the 16 highest, renormalize their weights to sum to 1. Two shared experts fire on every token, so a token normally touches 18. Those renormalized weights scale each expert’s contribution as it enters the residual stream.

The renormalization is where the second idea broke.

The three runs

Same flags throughout, full restart between each:

• RUN 1 — baseline. Top-k 16, weights renormalized over the 16.
• RUN 2 — top-32 pruned to 16. Select 32, normalize over 32, zero the 16 lowest. No renormalization after pruning, so the surviving weights sum to well under 1.
• RUN 3 — top-32. Weights normalized over the 32. 34 active experts per token instead of 18.

RUN 2’s missing renormalization is a confound I kept deliberately — renormalizing after the prune makes it near-identical to baseline by construction. The cost is that RUN 2 varies two things at once: wide-field scoring, and the total MoE contribution to the residual stream.

Implementation was an env-gated patch (K3_TOPK, K3_PRUNE_TO) to vLLM’s FusedMoERouter.select_experts, pruning via a post-selection topk plus scatter mask to stay CUDA-graph-safe, plus a num_experts_per_token override in kimi_k3/nvidia/model.py so buffers size to 32. Log markers confirmed the patch was live each run, and that pruning was off for RUN 3.

Results

lm-eval 0.4.12 against the local-completions API: gsm8k 5-shot (250 problems), arc_challenge 0-shot (300), mmlu (10 per subtask), humaneval (all 164).

What it says

Pruning without renormalizing is the worst of both worlds. RUN 2 loses or ties everywhere and drops 5.5 points on HumanEval. Wide-field scoring found no better 16. The likely cause is the confound above: with routed weights summing below 1, every MoE block contributes less than the model was trained to expect. That is an attenuated top-16, not a smarter one, and generation tasks — where error compounds across tokens — take the worst of it. Expert weight mass matters, not just expert identity.

The baseline holds on everything except code. Top-16 wins or ties on gsm8k, mmlu, and both arc metrics.

And then there’s HumanEval. RUN 3 moves it from 0.762 to 0.817 — +5.5 points on code, from a one-line routing change, at double the routed FLOPs per token.

Why I’m not calling that a win

That last number is the whole reason there’s a part 2.

Standard errors on these runs: arc ±0.027, mmlu ±0.011, gsm8k ±0.004–0.009, humaneval ±0.03. The +5.5 point HumanEval swing is about 1.8σ, single-seed on 164 problems. That is enough to justify a rerun and nothing else. Note also that RUN 2 posted a mirror-image −5.5 on the same benchmark, which should make anyone suspicious that HumanEval at this sample size is simply loose.

So the honest state after one afternoon: maybe wider routing helps code generation, maybe it’s noise, and the only way to tell them apart is to pay for more seeds. Two things worth running next:

1. HumanEval across several seeds, RUN 1 against RUN 3, on hardened variants with tighter error bars.
2. A look inside the router itself — before re-rolling dice, find out what the 896-expert distribution actually looks like. If routing mass is concentrated, wider selection should help; if it’s flat, the whole premise is shaky.

Both of those got run. Part 2 has the answer, and it is not the one this post is hoping for.

Setup and cost

Eight B300s (268 GB each) on a Verda spot instance in FIN-03 at $30/h. vLLM 0.27.1, TP=8, 8192 context, gpu_mem_util 0.92, FusedMoE path. The weights are 1.56 TB in BF16 across 96 shards.

Deploy to a serving endpoint took about three hours: seven minutes to boot, 1h40m to pull 1454 GiB of weights, 45 minutes to copy them onto a permanent volume, then a final half hour of bring-up including one restart for a missing ninja. RUN 1 took 42 minutes with detours for tokenizer and HF_ALLOW_CODE_EVAL fixes; RUN 2 and RUN 3 took 23 and 22 minutes once the pipeline was clean. Teardown, 10 minutes. Each eval set ran 25–40 minutes at ~32 concurrent requests.

Total ~$190, about 6.3 hours at $30/h. Two and a half of those hours went to moving weights.