No Individual GPU Operation Got Faster, but LLM Inference Did

Avi Chawla raised a counterintuitive point about LLM inference: even if every operation on the GPU takes exactly the same amount of time, inference can still get faster. The improvement does not come from the kernels themselves, but from the gaps between operations.

An LLM forward pass looks like a single computation. On the GPU, it is broken into a series of small operations such as attention, matrix multiplication, normalization, sampling, and data movement. The CPU prepares and submits these operations one by one. The startup overhead for a single launch is small, but during token-by-token decoding, a similar sequence is repeated for every token. Some kernels (GPU compute functions) take only a few microseconds, and the GPU finishes the current operation before the CPU submits the next one. The gaps left behind contain no model computation, and the profiler shows the GPU waiting for the next task.

CUDA Graphs are designed for this situation. They save an execution plan containing the sequence of GPU operations and their dependencies. The application first runs a warm-up, handles one-time initialization, and determines shapes, buffers, kernels, and memory addresses. At runtime, it captures the operation sequence as a graph. Subsequent iterations replay the entire graph with a single launch.

Replay is not reusing old outputs. New inputs are placed into the expected buffers, and the GPU re-runs the captured operations to produce new results. The computation changes very little; what changes is how the computation reaches the GPU.

The infographic above shows how warm-up, capture, and replay eliminate the repeated launch path.

The cost is flexibility. A captured graph requires compatible operations, tensor shapes, control flow, and memory addresses. Real inference traffic rarely has only one shape. Batch sizes change, requests come and go, and the server switches between prefill and decoding. Serving engines usually capture multiple graphs for common shapes, pad other requests to captured sizes, or fall back to eager execution.

The more shapes you capture, the higher the coverage, but GPU memory consumption and startup overhead also increase.

Some commenters pointed out that torch.compile(mode="reduce-overhead") in PyTorch does exactly this, automatically capturing CUDA Graphs. The effect of batch size is also obvious: at batch 1, decode performance is mainly limited by the number of operation launches, so CUDA Graphs provide a large benefit; with large batches, kernels run for longer, so the impact of the gaps becomes smaller. This is also why vLLM captures graphs for a fixed set of batch sizes.

The value of CUDA Graph replay lies in optimizing something that kernel benchmarks easily hide: the cost of repeatedly asking the GPU to start the next piece of work. This is one reason LLM inference needs a dedicated inference serving engine rather than a standard prediction API. Avi also wrote another article specifically about five architectural differences between traditional ML inference and LLM inference, including continuous batching, prefill-decode disaggregation, KV cache, prefix-aware routing, and model sharding.