GPU Kept Waiting for the CPU? Researchers Removed the CPU, Making Inference 8.47x Faster
Mainstream LLM inference frameworks have a hidden bottleneck: the CPU. The Blink architecture uses a SmartNIC to connect directly to GPU memory, removing the CPU from the inference path. P99 first-token latency drops to 1/8.47 of the original, generation speed improves 3.4x, energy consumption drops 48.6%, and it remains smooth even under CPU interference.
A tweet claims that Nvidia now has a big problem. The reason is that researchers removed the CPU from the LLM inference path, boosting inference performance by 8.47x.
This refers to the Blink architecture. Today's mainstream inference frameworks—vLLM, TensorRT-LLM, and SGLang—all share a hidden bottleneck: the CPU. The GPU is busy computing, but every request, every batch, and every token-level decision must first pass through the host CPU. Once the CPU gets busy with something else, GPU performance collapses. To keep CPUs from becoming overloaded, data centers have no choice but to leave large amounts of compute idle.
Blink uses a SmartNIC (smart network card) to receive requests and writes directly into GPU memory via RDMA (remote direct memory access); a kernel running continuously inside the GPU handles batching, scheduling, and KV-cache management itself. The CPU is not involved in the entire process.
The researchers ran comparisons on TensorRT-LLM, vLLM, and SGLang:
- P99 time-to-first-token (TTFT) latency cut to as little as 1/8.47 of the original
- P99 per-token generation latency (TPOT) cut to as little as 1/3.40 of the original
- Decode throughput improved by up to 2.1x
- Energy consumption per token reduced by up to 48.6%
Even more striking is the CPU interference test. When CPU load was artificially added, existing systems' performance dropped by up to two orders of magnitude, while Blink was almost unaffected.
The paper is by Mohammad Siavashi et al.; the preprint is at arXiv:2604.07609.
What is worth pondering is the judgment in the original tweet. Many commenters weren't convinced. Some users pointed out that Blink runs on Nvidia GPUs and BlueField DPUs and bypasses the host x86 CPU. The ones who should really be nervous are Intel and AMD; Nvidia may not be the loser—it might even be the winner.
If this approach is put into practice, data centers may no longer need to leave large amounts of compute idle for fear of CPU overload. The GPU can finally stop waiting for the CPU.
发布时间: 2026-11-20 02:10