SGLang Matches DeepInfer: What Benchmark Was the 1.04x Run On?

SGLang released a two-minute explanation and a comparison chart in response to a question: how did it match DeepInfer on DeepInfer's own benchmark.

After DeepSeek open-sourced DeepInfer, inference frameworks had a public, strong baseline. This time, SGLang ran DeepInfer's benchmark on the same set of hardware and reported two sets of results: prefill and decode.

First, the numbers. Prefill reaches up to 1.04x throughput on 4×GB300 and 4×B300. On GB300, the 4K point is about 1.04x, the 32K point is 1.035x, and the 64K point is 1.021x; 8K and 16K are essentially on par. On B300, the 4K point is about 1.039x, the 8K point is 1.025x, and the 32K point is 1.015x; 16K and 64K are near parity. A peak of 1.04x does not mean it wins at every point.

Decode is measured on 64×GB300, 16 nodes, DP64/EP64. Each request has a 128K context, of which 126,976 tokens are a shared cache prefix and 4,096 tokens are each request's own input. With 8 to 256 requests per GPU, SGLang's decode step time is 1.1% to 7.4% shorter in the 16-to-256-request range, and 0.4% longer at 8 requests. At 256 requests per GPU, DeepInfer takes 31.19 ms per step and SGLang takes 28.88 ms per step.

There is a detail that is easy to miss. The measured line in SGLang's chart is the measured result from running DeepInfer itself; the curve published by DeepSeek is the published result. Some commenters pointed out that SGLang's measured DeepInfer is already about 9% higher than the published curve at 4K and 8K, so the 1.04x is compared against a stronger measured baseline. If you compare directly against the README numbers, the 4K point would look higher. This explanation is more useful than just shouting a multiplier.

Where does the speed come from? SGLang integrated DeepSeek's FlashMLA, DeepGEMM, DeepEP, DeepSelect, and TileLang kernels into the main path, including MegaAttention, MegaMoE, MegaGate, and Mega-mHC. MegaMoE fuses token dispatch, expert GEMM, and combine into one kernel. Prefill uses breakable CUDA graphs covering up to the per-GPU 8K-token step, and decoder replay uses a separately sized graph. SWA bounded replay, DSpark speculative decoding, and both Engram placements—host memory and peer shard—are also in SGLang. zero-overhead scheduling has been there since v0.4, and DeepInfer's documentation also credits this design to SGLang.

SGLang also emphasizes that it is not just a collection of kernels. DeepInfer's README lists three pieces needed for a complete inference service but not included in the release: a KV cache system, prefill/decode disaggregation, and cross-replica load balancing. SGLang says it has all three. Its KV cache uses a unified radix cache; Linker connects the local L1 GPU cache to shared L3 storage such as 3FS or Mooncake, and HiCache supports host-memory L2. PD disaggregation supports Mooncake, NIXL, and MORI. The model gateway distributes requests across replicas and also supports routing by cache prefix.

The reproduction materials are detailed. The gist contains exact commits, dependency versions, launch commands, the benchmark client, and DeepInfer's Engram QP override script. Prefill corresponds to SGLang commit 7d996300b9, decode to 3b95120f00, and DeepInfer to 774ce24. The benchmark differences are also spelled out: DeepInfer's prefill measures about 9 batches on GPU 0, while SGLang averages complete batches across all GPUs in the 20% to 90% request-completion interval; DeepInfer repeats the same prompt on each GPU, while SGLang uses different new tokens per request. On the decode side, SGLang places the Engram table in device peer shards, while DeepInfer places it in host memory by default; DeepInfer's draft model has 64 redundant experts, while SGLang's draft does not have this option.

In the comments, some asked about more realistic scenarios: short prefill of 1K to 2K is very common in agents, and this set of charts does not cover it; others want to see throughput with mixed long and short requests, fixed TTFT/TPOT targets, and changes in cache hit rate. This time SGLang answers the same benchmark as DeepInfer; the full range of online scenarios is not yet covered by these numbers.

The most valuable part this time is laying out the baseline, reproduction scripts, and differences together. The 1.04x only applies to this benchmark set and cannot be extrapolated to all workloads. It is enough to show one thing: the optimizations released by DeepInfer have already entered SGLang's main path. Next, short prefill, mixed requests, and cache hit rate are more worth watching; those numbers are closer to the problems in production environments.