vLLM Says It Has Caught Up with DeepSeek’s In-House Engine—but Look Closely at the Footnote

The vLLM team published a blog post with a very direct title: on DeepSeek-V4.1-Flash, vLLM has caught up with DeepInfer.

DeepInfer is DeepSeek’s own inference engine, released only last week. A general-purpose engine benchmarking itself against a model vendor’s specialized engine was not initially expected to fare well.

First, the numbers. On the decode side, with a 131K context and nextn=3 configuration, for batch sizes from 8 to 256, vLLM’s throughput on GB200 is within 6% of the curve DeepInfer published. Moreover, vLLM uses DEP32, while DeepInfer uses EP64DP64—a twofold difference in parallelism.

On the prefill side, vLLM’s DEP2 curve also tracks the curve DeepInfer published. But there is a footnote: that DeepInfer data set is EP4DP4, 32 redundant experts, 140-token bounded replay, and the hardware is not specified; vLLM is GB300 DEP2, no redundant experts, 128-token replay, and its own prompts. The setups differ, and vLLM itself says in the blog post that this gap cannot be read as an engine-to-engine speedup.

The tweet says “match”; the blog says “not an engine comparison.” Both framings coexist. Keep that in mind when reading.

What is really interesting is the lineage of the two sides. DeepInfer was originally modified from vLLM, and later vLLM took back the kernels open-sourced by DeepSeek—MegaAttn, MegaMoE, and Mega-mHC are all in there. One online commenter summed it up quite accurately: fork it out, optimize internally, open-source the kernels, and then upstream catches up. The open-source cycle has come full circle.

The numbers at scale are even harder. On a single GB300 NVL72 rack, with P/D disaggregation plus Wide-EP, running SemiAnalysis’s AgentX, it reaches 954K tokens/s per GPU, with single-user P90 at 196 tokens/s—7.9× the single-node result. At DeepSeek’s off-peak pricing, that is $38.96 per GPU-hour.

Half the credit for these numbers goes to the model itself. V4.1-Flash’s global KV cache is only 890 bytes per token, thanks to cross-layer KV sharing plus FP4 compression. As a result, context was stretched from 128K to 855K, a 6.7× increase, while decode latency rose by only 0.6 ms, or 6.7%.

So prefill does not need large parallelism. In actual tests, DEP4 delivers only 83% of DEP2’s per-GPU throughput—53.3K versus 64.4K tokens/s. The cause is not the kernel but uneven DP load; when ranks are out of sync, DEP4 drops by 13.6%. DEP2 became the default option.

Prefill also revealed three bottlenecks. Layer 20’s SWA bounded replay extended to all layers: +19.9% at 32K new tokens; the Engram host table switching to 2MB pages: +2.8%; the prefill indexer using DeepSelect top-k: +15.6%. Stacked together, after a 128K cached prefix, throughput for 4K to 64K new tokens improves by 35.6% to 43.9%, rising from 71.2K to 102.5K tokens/s/GPU at 64K.

The decode side uses Wide-EP. From DEP4 to DEP32, with fewer experts per rank, MegaMoE time drops from 13.4 to 6.0 ms, while the remaining work stabilizes at around 10 ms, showing diminishing returns. EPLB rearranges experts every 3,000 steps, pulling target balancedness from 0.35 to 0.94 and draft from 0.21 to 0.89, essentially catching up with uniform routing.

The easiest part to overlook is the CPU section. Unoptimized DEP32 takes 17.57 ms per step, slower than DEP16’s 13.84 ms, even though its GPU execution is actually 1.1 ms faster. Wide-EP requires all-rank synchronization at every layer, so one slow rank drags down the entire instance. KV reads were changed to NIXL push, Gloo all-reduce was replaced with recursive doubling, and the number of new KV loads per step was limited. Altogether, this took it from 141K to 440K tokens/s per GPU, with P90 from 40 to 159. No matter how fast the kernel is, there is no use if the host is sitting there waiting.

There is also a dedicated decode path for small batches. With four B300s, single-user P90 is 826 tokens/s, versus 575 on the standard path—44% higher.

Back to the beginning. The fact that a general-purpose engine has caught up with a specialized engine itself shows that the gap is not about who understands the model better; it is about who is willing to grind through engineering details to the end. DeepSeek’s open-source kernels and DeepInfer’s curves gave vLLM references and a foundation, and this cycle benefits both sides.

Just remember, when looking at comparison numbers, to distinguish which parts are read off a chart and which are actual runs, and how much the hardware and parallelism differ. Tweets can be short; footnotes are often longer.