SGLang Ran DeepInfer's Benchmark and Wins at the System Layer

SGLang Says It Has Caught Up; First, the Numbers

After DeepSeek open-sourced DeepInfer, SGLang ran DeepInfer's own benchmark. Conclusion: prefill throughput is up to 1.04x on 4x GB300 and 4x B300; decode on 64x GB300 has step times 1.1% to 7.4% shorter than DeepInfer at 16 to 256 requests/GPU, and 0.4% longer at 8 requests/GPU. The remaining points are within 1%.

The figure's three subplots correspond to 4x GB300 prefill, 4x B300 prefill, and 64x GB300 decode, respectively. Note that prefill and decode are best-case scenarios at two hardware scales, not a single end-to-end run. Some users pointed this out; SGLang's tweet used “both,” but the numbers come from two sets of machines.

The Reproduction Guide Is More Useful Than the Peak

The reproduction gist provides the commit, package versions, model revision, launch scripts, and benchmark scripts. Prefill uses 7d996300b9, decode uses 3b95120f00, and DeepInfer uses 774ce24. Even DeepInfer's Engram QP settings are patched via sitecustomize.py.

SGLang's prefill metric takes complete steps across all ranks between 20% and 90% request completion; DeepInfer takes about 9 steps from rank0. Decode uses SGLANG_SIMULATE_ACC_LEN=3.25 to simulate acceptance length, while DeepInfer uses real acceptance. The two sides each differ in one place regarding Engram placement and draft redundant experts. The reproduction guide spells out these differences, which is more useful than reporting only a peak.

Where the Speed Comes From

SGLang integrates DeepSeek's FlashMLA, DeepGEMM, DeepEP, DeepSelect, TileLang kernels, and the Mega series. MegaMoE fuses token dispatch, expert GEMM, and combine into a single kernel; MegaAttention is the default attention path on B300/GB300. DSpark speculative decoding proposes multiple tokens at once, and the target model verifies them together. SWA bounded replay makes the decoder replay only the last 128 tokens, rebuilding the missing sliding-window KV when the prefix cache hits. Engram tables can live in host memory or be split across GPUs and read directly from peers over NVLink. SGLang also says prefill uses breakable CUDA graphs, covering prefill steps up to 8K tokens/GPU.

The full technical blog covers kernel fusion, parallelization strategy, and benchmark setup in some detail.

The System Layer Is SGLang's Real Strength

DeepInfer's README lists three components needed for a complete inference service but missing from the release: a KV cache system, prefill/decode disaggregation, and cross-replica load balancing. SGLang has all of them. Its radix cache reuses KV across requests and conversation turns; Linker connects the L1 GPU cache to L3 shared storage; HiCache has a host-memory L2 tier; P/D disaggregation supports Mooncake, NIXL, and MORI for KV transfer; and the model gateway routes by cache prefix.

AGTP's summary chart puts vLLM and SGLang side by side, with the headline “Caught Up in a Few Days.” The chart is good for a quick look, but the two numbers come from different hardware scales; don't treat the 4-GPU prefill and 64-GPU decode as one run.

Assessment

SGLang's win this time is in system integration, not a victory of a single kernel. DeepSeek open-sourced the kernels and a reference engine; SGLang plugged the kernels into a production-capable serving stack and then validated it with the same benchmark. That sequence is more meaningful than “another record broken.”

The questions worth keeping are also there: short prefill under real agent workloads, mixed long and short requests, fixed first-token and single-token latency targets, and throughput after cache hit rates change—these are closer to production than a 4K to 64K sweep. Some users asked about 1K to 2K short prefill, and they are right to ask. Others in the comments asked about H20 support; the source material has no answer.

The lead time for inference engines is now measured in afternoons. Today SGLang measured itself with DeepInfer's yardstick; tomorrow the yardstick may change hands again.