vLLM Hits 58x Throughput, 387 tokens/s Per User—Are Tokens About to Become Oversupplied?

vLLM published a set of numbers on X: DeepSeek-V4.1-Flash running on NVIDIA Vera Rubin NVL72 reaches 1.7 million tokens/s per GPU at P90 387 tokens/s per user, 58x that of GB300 NVL72. Peak per-GPU throughput is 2.52 million tokens/s, and at single-concurrency batch size 1, P90 is 1810 tokens/s. The benchmark is SemiAnalysis AgentX, configured with P/D disaggregation and Wide-EP. P/D disaggregation means prefill and decode are deployed separately, while Wide-EP means expert parallelism is spread more widely.

The numbers are clear:

• Peak throughput 2.52 million tokens/s/GPU, 2.7x GB300.
• At P90 387 tokens/s/user, 1.7 million tokens/s/GPU, 58x GB300.
• Single concurrency batch size 1, P90 1810 tokens/s, using TP8.
• Throughput counts cached input, uncached input, and output tokens.

The 58x comes with caveats. If you change the benchmark, parallel strategy, or hardware, the numbers cannot be carried over directly. The official post gives the measurement method: use vllm bench forward through the real execute_model path, with real weights and AgentX tokens; run kernel A/B loops for several seconds and take sustained values.

Three hardware features of Rubin determined most of the optimization direction. First, each GPU has two memory domains; an SM reads its own domain quickly and the other domain slowly. Bandwidth-sensitive data stays in the local domain: MoE weights, LM head, MegaMoE combine buffer, and mHC residual. Second, SM107 adds wider block-scaled MMA. K64 FP8×FP4 does twice the work at once, used in MegaMoE and low-latency routed GEMM; K128 FP4 MMA is used in the indexer, making decode MQA logits 15% to 22% faster. Third, 72 GPUs share one NVLink domain. TP8 spans two compute trays, all-reduce costs only 0.3 microseconds more, and per-GPU weight reads are almost halved.

vLLM retains the three engine forms from GB300: DEP2 prefill, Wide-EP decode, and low-latency TP decode. DeepSeek's kernel set is also retained, with MegaMoE, MegaAttn, Mega-mHC, MegaGate, and DeepSelect rewritten for SM107.

DEP2 prefill step is about 28% faster. Mega-mHC keeps residual in die-local memory; MegaAttn lets tensor cores avoid waiting on loads; MegaGate only computes exact sqrt-softplus for experts that still have a chance to enter the top 6—about 7 of 384 experts pass. MegaMoE sizes tiles according to the token count of the busiest rank, reducing time by 55% in a 40-step replay.

DEP decode step is about 25% faster. MoE weights are split by output column across the two memory domains, and SMs in each domain read only local tiles; Wide-EP dispatch does one NVLink round trip per token; combine uses a sentinel to wait for data rather than relying on a barrier; split-K GEMM spreads K across the SM107 cluster.

Low-latency TP8 decode spans two trays, with each rank holding 288 columns of experts and 8 attention heads. The official post claims P90 1810 tokens/s at batch size 1, with P50 rising from 1117 to 1844 tokens/s/user. Techniques include packed FP4 weight stages increasing shared memory from 7 stages to 11 stages, saving 57 microseconds per step; shared experts packed into routed GEMM, saving 220 microseconds per step when below 16 rows; all-reduce and mHC fused into one pass; sparse MLA gathering KV in advance.

Rubin computes quickly, so a single rack serves more sessions at once, and the live prefix exceeds single-rank cache. vLLM pools rack memory: the Mooncake KV pool spans DRAM and spare HBM over RDMA; Engram's two 94.4 GiB hashed n-gram tables are sharded across GPU HBM over NVLink, so each prefill instance no longer keeps a copy in host memory. Under high concurrency, the prefix hit rate is 96.5% to 97%. Dynamo routing sends requests to the worker and DP rank with the least uncached work. With three DEP2 instances at concurrency 4800, request count is 68% higher, TTFT P50 drops from 29.8 seconds to 4.2 seconds, and hit rate rises from 91.8% to 96.1%. In 12 DEP2 simulations, rank idle time drops from 10.3% to 1.3%, and TTFT P90 goes from 0.98 seconds to 0.38 seconds.

Some commenters asked how concurrency changes the curve; some said 1810 tokens/s would scroll too fast for a terminal to keep up; others think token oversupply will appear in 2027–2028. @阿哲聊芯片's observation is worth noting: listing total throughput and per-user speed together is more intuitive than reporting only peak. Looking only at peak easily ignores per-user speed under concurrency; this time the official post puts P90 387 tokens/s and 1.7 million tokens/s/GPU together, which carries more information than the 58x alone. Whether tokens become oversupplied depends on how application-side consumption changes after efficiency gains, and on whether these optimizations can move from a specific benchmark to general workloads.

Original article: https://vllm.ai/blog/2026-11-12-deepseek-v41-flash-vera-rubin
X post: https://x.com/vllm_project/status/2121008091538092393