230 Experiments, 94 Valid: Agents Turned LLM Inference Optimization into a Pipeline

Yangqing Jia posted a hand-written analysis on X about using agent fleets to optimize LLM inference. The data he gave comes from optimizing their own DeepSeek 4.1 Flash: 2.41 ms per step, with Dspark generation speeds reaching up to 2,300 tok/s, while agent tasks held steady at around 1,800 tok/s. The agent ran about 230 optimization proposals, and 94 were validated as effective.

This is worth looking at because most discussion is still stuck at the kernel level. Yangqing Jia's conclusion is that there is still a lot of room in the end-to-end system. Below are his 8 observations, plus a few additions from the comments.

1. The gains are not only from kernel optimization. More than 50% of the optimizations did not involve new kernels; they came from configuration, computation graphs, launch order, and other strategies. Kernels have already been heavily explored, but there are still many gains to capture in other parts of the end-to-end system.

2. MegaKernel may be overrated. On non-small models, MegaKernel is trying to swallow too much at once. In most of their experiments, the biggest gains came from gluing small-block operators into existing kernels, or fusing multiple small kernels into medium-sized kernels. This conclusion may only apply to NVIDIA GPUs, because the underlying drivers are already highly optimized; vLLM recently also got good results using megakernels on TPU.

3. KernelBench and real serving are different problems. Many kernels show positive gains in isolated tests but become negative returns when put into an end-to-end composition. Yangqing Jia has always wanted a simulator like Accel-Sim, but current simulators are not accurate enough for complex latency patterns that mix compute and memory.

4. Not wanting to be chained to a workstation makes validation very important. As long as you can define what correct means, such as numerical tests and end-to-end quality evaluation, agents can run for a long time. They did not intervene in the runs; they only did three things: add GPU machines, make sure the token limit was not hit, and raise the credit card limit.

5. Convergence is not a concern. He believes the community already has a consensus: once the objective is set, the agent will eventually converge. The key becomes convergence speed, and the credit card limit in point 4 is very real.

6. At low concurrency, low-level assembly techniques such as SASS editing become less effective. The main bottleneck becomes waiting on memory and caches.

7. The hot path has been partly solved, but there is still more to mine. He believes that in the coming months, larger gains will come from wider distributed serving stacks and system optimizations beyond a single serving engine. This will involve a lot of systems and SRE experience: setting up telemetry, running A/B tests that do not disrupt production traffic, similar to what recommender systems did back then.

8. Agents doing end-to-end optimization is a necessity. 230 experiments, each potentially costing a person a week; even with manual experiments plus Claude Code, it would take at least several days. At this scale and speed, optimization directions that human bandwidth could not originally cover got explored.

In the comments, Quantized asked how many optimizations came from batching or scheduling changes. Yangqing Jia replied: about 20 from scheduling, and about 5 from making kernel launches more programmatic, such as reordering streams and stream dependencies. The categories were self-reported by the agents, but after manual spot checks, he considered the classification fairly accurate. Aleksandar Janca said point 4 also holds outside inference: the ceiling of an agent fleet depends on checks that judge correctness. Lei Wang added: LLMs are very good at creating tech, but not yet at creating science. Science is based on a smaller rule set; tech is searching in infinite combinations.

Yangqing Jia ended with a contradictory judgment: currently, agents are more like good engineers. They can execute known insights at scale, interpolating faster. He has not yet seen agents design algorithms like PagedAttention or Winograd. But in the coming months, extrapolating beyond the existing knowledge base may not be impossible.

The most notable thing in this set of results is that the object of optimization has shifted from single-point kernels to end-to-end systems. If you only watch kernels, you may miss more than half of the gains. Another judgment is that agents currently look more like high-speed executors, able to run known insights at a scale humans cannot keep up with; they have not yet proven they can propose new algorithms of the same magnitude. This boundary is worth continuing to watch.

Original link: Yangqing Jia's X post