Benchmarking Trillion-Parameter Open Models Without Fooling Yourself

We ran Kimi K3 and DeepSeek V4 (Pro and Flash) head-to-head on a hard-benchmark suite — same harness, same serving stack, same hardware, back to back. The scores are below, but the scores are honestly the less useful half of this post. The useful half is the list of ways the numbers were nearly wrong.

The setup

Everything ran on one 8×B300 node (tensor-parallel 8) with vLLM, driven by lm-eval plus custom harnesses for AIME-25 and BBEH. Identical prompts, budgets, and scoring for all three models. We deliberately dropped saturated benchmarks (GSM8K, MMLU, ARC — everything scores in the high 90s or clusters within noise) and kept rows with headroom:

K3 leads every row on this harness, with the largest gaps on AIME-25 (+33pt) and BBEH. And BBEH turned out to be exactly the benchmark we wanted: the best model scores 30%, so there’s real signal and no ceiling for the next model generation to hide against.

Now, the traps.

Trap 1: the chat template was silently thinking

The biggest scoring bug of the project. K3’s chat template enables a structured thinking channel by default — and the serving stack had no parser for it, so the model’s reasoning landed in the content field, glued to its answers.

For strict-format benchmarks this is fatal: on an IFEval probe, K3 scored 0.15 before the fix and 0.90 after — the same model, the same questions, six times the score. The fix was two lines (disable thinking via a chat-template kwarg, add the proper stop token), but nothing errored before it. The numbers were just quietly garbage.

If you benchmark a new open model, read its chat template before you trust a single number. Thinking-by-default is becoming common, parser support is not.

Trap 2: the scorer can be more brittle than the model

On MATH-500, the same model outputs scored 0.26 with lm-eval’s exact-match metric and 0.97 with math_verify. The difference is entirely LaTeX formatting — \frac{1}{2} vs 1/2 vs 0.5. Exact-match on free-form math is a formatting benchmark wearing a math costume. Use a semantic verifier or don’t report the row.

Trap 3: harness compatibility masquerades as model quality

In an earlier same-harness pass, DeepSeek V4-Pro appeared to score 0.16–0.33 on HumanEval — absurd for a frontier model that reports high-90s on code. Sample inspection showed why: the model emits a reasoning prelude that swallows the code block’s opening, and lm-eval’s answer-extraction filter can never assemble a valid function. That’s not a model deficiency; it’s a harness incompatibility. We marked the cell n/a rather than publish it. The general rule: a shockingly low score is a bug report about your harness until proven otherwise.

Trap 4: your own interim numbers

Halfway through the overnight run, interim status messages cited two BBEH figures that, at final compile time, traced back to no artifact on disk — one came from a run that was later invalidated, one apparently from nowhere. They were caught because the write-up rule was mechanical: every published number must point to a result file. Both got corrected before anything shipped. Adopt the rule; it’s embarrassing exactly once.

The honest caveats on the table above

• K3 ran with thinking disabled (see Trap 1 — no parser means thinking-mode output can’t be scored cleanly). Its numbers are therefore a floor: K3 with a working reasoning budget is untested here and can only be higher.
• The DeepSeek models reason inline by default within the same token budgets, so “same budget” ≠ “same compute per answer.” Style differences leak into every cross-family comparison; ours is no exception.
• BBEH-mini is a seeded 230-question sample with ~±3pt stderr: K3’s lead is real, but Pro-vs-Flash on that row (10.4 vs 13.0) is a coin flip.

What it cost

The whole suite — three models served serially, five benchmarks, plus the diagnosis and rerun cycles the thinking bug forced — took about 11 hours on one rented 8×B300 node on Verda. The earlier same-harness comparison round was another ~9. Frontier-scale eval work is no longer a lab-budget activity; being wrong in ways you don’t notice is the expensive part.