Nearly 1,000 tok/s/user—yet SGLang gets asked about batch size first

The official SGLang account posted a performance update: DeepSeek V4.1 Flash on GB300 NVL72, using SGLang's aggressive kernel fusion, reaches nearly 1,000 tok/s/user in single-user throughput. The benchmark comes from SemiAnalysis, and SGLang also thanked contributors and NVIDIA.

SemiAnalysis's original tweet puts it more bluntly: SGLang on GB300 NVL72 can push DeepSeek Flash v4.1 to nearly 1,000 tok/s/user by aggressively fusing kernels together. The tweet also notes that TileRT kicked off this trend in the megakernel engine ecosystem for open-source frontier models, and SGLang is adopting it widely.

Kernel fusion sounds like low-level optimization, but the practical effect is easy to grasp. Someone in the replies summed it up: fewer kernel launches and less intermediate data detouring through GPU memory mean throughput goes up. For a serving stack, this kind of optimization can squeeze out extra room without changing hardware, which is why the number lands so loudly.

But the replies are closer to the engineering reality than the number is. The Daily Inference asked first: what is the batch size? Mark followed up: does it still hold at large batch sizes, or only for a single user? Stephen J Ronan MD cared about another boundary: are these fusions all done per GPU, or do they depend on the NVL72 domain? People running the model on a single GB300 want to know whether they can keep part of the gains. Wolfgang asked directly how much latency improved.

Precision did not escape scrutiny either. Aapakari asked whether the 552B figure on GB300 is FP8 or FP4. Atlas was more positive: getting to nearly 1,000 tok/s/user through kernel fusion alone shows the serving stack still has headroom, and it is not just about hardware. Semiconductor Bull was more cautious, arguing that the software jump on the same GB300 is impressive, but there are only so many more times this kind of headroom can be mined before hitting a ceiling.

Anton Liubakhinets's quip sounds most like production: at 1,000 tok/s, the model finishes answering before a person has finished reading the question, and in the end only the review queue still moves at human speed.

This news is worth reading, but don't rush to treat single-user throughput as a server-side conclusion. The two numbers most worth waiting for next are the corresponding batch size and the latency improvement. The original benchmark is in SemiAnalysis's tweet, and SGLang's original tweet includes an image as well.

Original tweet link: https://x.com/sgl_project/status/2115913588741468629

SemiAnalysis link: https://x.com/SemiAnalysis_/status/2115821309137465536