Agent Loop Is Like SGD Without Gradients; DeepSeek's Latest Optimization Round Is Down to 2.32 ms/step

Yangqing Jia likened the agent loop to stochastic gradient descent in a tweet, adding: without gradients. That analogy becomes concrete when applied to DeepSeek's optimization data: after one night of optimization, DeepSeek-V4.1-Flash ran at 2.32 ms/step on 8×B300, below the previous empirical lower bound of 2.5 ms/step, and it was still improving. He had an AI draw the statistics, and several observations are worth noting.

1. The agent was not asked to do this, but empirically it captured early gains first, then improvement slowed.
2. No experimental arrangement was made to separate independent contributions; by eye, later experiments had smaller increments.
3. Convergence looks consistent, unlike the agent running out of ideas.
4. The experiment acceptance rate was very high at first, then stabilized around 46%.
5. The gains look more like exponential decay than a heavy tail.

In the chart, the horizontal axis is the number of experiment runs, 0 to 300; the vertical axis is milliseconds per step, 2.00 to 3.75. The blue exponential fit gives a floor of 2.12 ms, τ=119; stock vLLM TP8 is 5.654 ms. The 2.32 ms/step point is still on the descending part of the curve.

The cumulative acceptance rate reaches 46.2% at about 108 experiments, i.e. 108/234, and then is basically flat. A high early acceptance rate is intuitive; what is remarkable is that it did not keep collapsing. Rejected samples account for more than half, and the information is mainly hidden in that rejected half.

SGD has improved over the years through rigorous mathematical explorations such as momentum, hessian-free, ADAM, and Muon. The agent loop faces discrete optimization: no coordinates, no gradients, a non-fixed landscape, and the loop is more like sample, reject_or_accept, resample. The best heuristic and algorithm are still undetermined. Yangqing Jia thinks it resembles Kalman filtering, rejection sampling, and data-driven Markov-chain Monte Carlo, but the precise mathematics is unclear. He looks forward to larger experiments and more traces, with engineering and research tied together.

Daniel Lenton asked in a reply whether, before proposing the next experiment, the agent reads the previous 126 rejects. That pile of rejected samples is the thing in the loop that most resembles a gradient. Allen asked whether the key is to get the agent going in the right direction from the start. Both questions point to the same place: whether the reject pool enters the next round of decision-making. If it does, the loop has memory; if it does not, the loop is more like blind sampling. A 46% acceptance rate means rejected samples are the majority, and the clues for improvement are probably buried there too.

Original: https://x.com/jiayq/status/2116201483196887199