1M-token prefill in about 12 seconds: Baseten open-sources DeepSeek inference engine

Baseten's Phil Howes has open-sourced the inference engine for DeepSeek-V4.1-Flash. The core engine is Rust + CUDA, with no PyTorch, SGLang, or DSL. The numbers in the GitHub table are: 1,740 tokens/s decoding on 4-card GB300 and 2,060 tokens/s on 8-card B300; prefilling 1 million tokens takes 13.2 seconds and 11.7 seconds, respectively. In his tweet, Phil Howes wrote about 1,740 TPS on 4xB300 and 2,060 TPS on 8xB300. TPS is tokens per second, and a token can be roughly understood as a small chunk of model output text.

The repository is called deepseek-ultrafast; it supports a 1M context length and also supports multi-server KV cache sharing. The deployment requirements are clearly stated: 4-card GB300, 4- or 8-card B300, 4- or 8-card B200, the CUDA 13 toolchain, and Redis for multi-machine KV cache coordination. The model is already available on Baseten Model APIs.

Beyond performance, multi-machine KV cache sharing is the more practical part of this release. Each server keeps the KV cache for recent conversations in host memory. When the next round of requests lands on another server, that server pulls it directly from the machine holding the cache via RDMA, host memory to host memory, without recomputing the prompt. Redis records which server owns each conversation's cache and broadcasts ownership changes. The README says this setup removes the need for prefix-aware routing, making simple purely load-aware routing SOTA. For deployment teams, this is closer to everyday problems than a pretty decode number.

There is one more detail worth noting. Phil Howes replied that the team merged the results with others' work and incorporated some soon-to-be-open-sourced SGLang work; mahmoud turned it into a complete inference server that shares KV cache across RDMA. The README acknowledgements also say that the kernels were optimized by coding agents capable of reading FlashInfer and SGLang source code. The core engine does not use SGLang, but the coding agents optimizing the kernels read the SGLang and FlashInfer source code. The boundaries of the open-source world are sometimes blurrier than slogans.

When Sarah Guo reposted it, she wrote: the future is generated, continually learning inference engines in open source. The source material does not elaborate on the implementation of 'continually learning.' Some commenters asked whether this means weights will change at serving time. Others asked about batch size, request mix ratio, and per-request latency and TTFT when long prefills and short follow-up questions arrive at the same time. These questions are not answered in the source material.

What can be confirmed now is that the code is public, the model API is available, and the performance numbers and reproduction scripts are all in the repository. As for how much of these numbers hold up in real interactive coding-agent scenarios, we'll have to wait for more request patterns to be run.