vLLM closes to within 6% of DeepInfer, with a reminder tucked into the footnote

vLLM posted an official update: on DeepSeek-V4.1-Flash, vLLM's decode throughput is now within 6% of DeepInfer. The official tweet's comparison is DEP32 versus DeepInfer's EP64DP64. DeepInfer is DeepSeek's in-house inference engine, originally based on vLLM. Now vLLM has integrated DeepSeek's open-source kernels and added prefill and decode optimizations, bringing the two performance curves closer.

The chart comes from the vLLM tweet. Decode is on the left, Prefill on the right. The two lines look very close, but before reading this chart you first need to look at the comparison conditions.

How the 6% figure was derived

The blog is clear: for decode it uses DeepSeek's configuration, 131K context, nextn=3, batch size 8 to 256. On GB200, vLLM's per-GPU output throughput is within 6% of the result DeepInfer published. Someone in the comments pressed on the comparison basis: DEP32 has only half the ranks of EP64DP64; if matched by total volume, per-GPU would be about 2x. Official Figure 2 is labeled as per-GPU output throughput, so readers can check against the chart.

The Prefill comparison needs even more attention. The prefill curve DeepInfer published uses EP4DP4, 32 redundant experts, 140 token replay, and does not specify the hardware; vLLM uses DEP2, no redundant experts, 128 token replay, and its own prompts. The blog explicitly says the settings are different and does not treat the gap between the two lines as an engine-to-engine speedup. X user ren pointed out that DeepInfer's numbers were read from the README chart and not rerun; the prefill settings differ, and the blog also does not count the gap as an engine-to-engine speedup. X user Aapakari found this somewhat bold: the footnote says it is not an engine-to-engine comparison, while the main text uses the word 'match'.

954K requires an entire rack

vLLM ran SemiAnalysis AgentX on GB300 NVL72, using P/D disaggregation and Wide-EP. The result was 954K tokens/s per GPU and P90 196 tokens/s per user, 7.9x the single-node result. At DeepSeek off-peak pricing, that converts to $38.96 per GPU-hour. There is also a dedicated decode path for small batches: on four B300s, a single user gets P90 826 tokens/s, 44% higher than the standard path's 575.

X user Mark Ellis noted that 954K requires a full rack: P/D disaggregation plus Wide-EP, and the 7.9x is relative to a single node. What self-hosting teams can actually verify is still the single-node numbers and CPU overhead.

Measure the model first, then discuss optimization

The official post first measured DeepSeek-V4.1-Flash's architecture: global KV cache is only 890 bytes per token, thanks to cross-layer KV sharing and FP4 compression. Layers 2, 8, and 14 use 2x compression for full-context indexing; layer 20 writes the global KV for all tokens, but indexing and attention run only over the last 128 tokens; layers 21 through 39 process only those tokens. As a result, from 128K to 855K—a 6.7x increase in context—the target model's decode latency increases by only 0.6ms, a 7% rise.

Even with long context, decode is mainly limited by MoE: with DEP4, 128K, and 64 requests per rank, MoE accounts for 62% of the target step, while attention and indexer together account for 13%. So Wide-EP directly cuts weight reads and is effective for decode.

Prefill is more stable with DEP2

For AgentX prefill-only at concurrency 256, DEP4's per-GPU throughput is only 83% of DEP2's, 53.3K versus 64.4K tokens/s. Mirror and lag replay show that when ranks are out of sync, DEP4 loses 13.6% throughput and DEP2 loses 5.9%. DEP2 became the simple default.

Three changes stack up: expanding layer 20 to bounded replay gives a 19.9% improvement at 32K new tokens; switching the Engram host table to 2MB pages gives 2.8%; using DeepSelect top-k for the prefill indexer gives 15.6%. Combined, that is a 35.6% to 43.9% improvement across 4K to 64K new tokens, 42.5% combined at 32K, and at 64K from 71.2K to 102.5K tokens/s/GPU, a 43.9% improvement. All these numbers are based on a 128K cached prefix.

The hidden trap in Decode is the CPU

Wide-EP scaling from DEP4 to DEP32, with 64 requests per GPU and 128K context, cuts target MegaMoE from 13.4ms to 6.0ms, leaving about 10ms of remaining work, and gains start to diminish. At DEP32, draft communication overhead also offsets part of the gain.

EPLB rebalances experts every 3000 steps, using the load from the first 1000 steps. Balancedness for the target model rises from 0.35 to 0.94, and for draft from 0.21 to 0.89, recovering 97% and 93% of the gap to uniform routing, respectively. PR #62009 uses the NVLink symm_mem backend; DEP32 rebalancing takes under 1 second and does not stop decode; it folds replica mapping and load recording into MegaMoE, saving about 0.2ms per step; and it removes host synchronization from the decode loop. #61395 supports heterogeneous target and draft models—DSpark draft has 128 experts with top-3 routing, the target has 384 with top-6, and EPLB tracks and rebalances them separately.

The most counterintuitive part is here: unoptimized DEP32 takes 17.57ms per step, slower than DEP16's 13.84ms, even though GPU execution is 1.1ms faster. Wide-EP synchronizes all ranks at every layer, and one slow rank holds back the instance; the more ranks there are, the shorter the GPU step and the less CPU headroom remains. The fixes are concrete: NIXL push plus UCX scatter-gather packs KV rows for sending, taking throughput from 141K to 440K tokens/s/GPU and P90 per user from about 40 to 159; recursive doubling CPU all-reduce cuts DP synchronization from 10.8ms to 1.5ms per step and the step from 16.10ms to 13.15ms; admission allows at most 4 new KV loads per step, taking throughput from 460K to 516K tokens/s/GPU and P90 per user from 109 to 153; prompt admission uses an int32 buffer, skips empty sliding-window blocks, and uses chunked copy, cutting CPU admission for an 855K prompt from 38.8ms to 6.0ms.

X user AxleMode said the CPU section is the sleeper: DEP32 does less GPU work but serves more slowly, and fast kernels cannot save every rank waiting on the host.

The open-source loop

DeepInfer was originally based on vLLM; DeepSeek open-sourced the kernels; vLLM then brought back MegaAttn, MegaMoE, Mega-mHC, and others, catching up to DeepSeek's own engine. X user Michal Oblak summed it up as: fork an open-source engine, optimize internally, open-source the kernels, and watch upstream catch up. X user ShadowAguy said 'full circle'.

The comments are still asking: can the prefill improvements be transferred to other MoE models, or are they only effective for DeepSeek-V4.1-Flash's architecture? The official post does not answer. Self-hosting teams should verify the single-node numbers and CPU overhead; for the 954K figure, look at the rack first.

Engineering blog: https://vllm.ai/blog/2026-10-23-deepseek-v41-flash-pd-wide-ep