AI tunes its own inference stack: 14 PRs merged into SGLang-Omni, with gains up to 344%

OpenRSI and the SGLang-Omni team have applied Recursive Self-Improvement to production AI infrastructure. The result is not just a concept blog post: 8 speech and omni models, 14 PRs reviewed by SGLang-Omni maintainers and merged upstream, with throughput improvements from 7.6% to 344.3%. Up to +326.5% on H100 and up to +344.3% on H200. The highest numbers come with caveats, explained below.

The term RSI has been used loosely lately. Believers and skeptics have already diverged in their judgments about how AI will make money. OpenRSI has now offered a version that can be checked: agents improve the infrastructure serving AI models, and each verified and merged change becomes the baseline for the next round. Humans define the problem, system design, and review; agents handle the experiment-intensive part of the work.

There is one detail from the tweet worth putting up front: the highest 300%-level improvements happened because this model was handed to the OpenRSI team two days after the support process began and had not yet been manually optimized, so the gain was especially large. Most models that saw clear improvements gained around 30%. 30% is still significant, but it is not the same thing as 344%.

The method is called Harnessed RSI. The common approach today is for engineers to drive coding agents: humans choose experiments, read results, and decide what to do next. The human is the loop. Harnessed RSI removes humans from the inner loop of performance optimization, but does not remove humans from the work. Humans write proposals and define the task, benchmark, baseline, and verifier. Agents change code, run experiments, and measure in the inner loop. RSI-Harness manages the outer loop, deciding when to score and carrying lessons from each round into the next. An independent verifier cannot see the agent’s speculative path and only accepts reproduction. Merged changes become the new baseline.

The process is called RSI-Anything: a human writes a proposal, an agent turns it into an executable benchmark and evaluator, then iterates, reproduces, and opens a PR. SGLang-Omni maintainers review, sometimes adding code and sometimes reproducing it themselves. The blog says review took a median of about 3.5 days, with up to 23 commits. The bottleneck has shifted from “experts do not have time to run experiments” to “review bandwidth and evaluator quality.”

The 14 PRs fall roughly into five categories, all aimed at runtime overhead, without changing model weights or sampling settings. First, idle threads were spinning and competing for CPU cores; they are now put to sleep, returning cores to the pipeline. Second, tensors crossing stages were converted to Python lists and rebuilt; now tensors stay tensors, with typed bytes only in the final message. Third, computations no one uses are skipped: unneeded quantizer reconstruction, duplicated frontend work, and audio feature extraction beyond what is needed. Fourth, too many small GPU launches are replaced by CUDA graphs or fused kernels replayed once. Fifth, prefill and decode were waiting on each other across two machines; now the prefill machine processes the next request early, and the decode machine picks it up when free.

Look at a few models for specific data. CosyVoice3 on H100 non-streaming/streaming improved +44.4%/+46.2%, and on H200 +48.1%/+56.3%. MOSS-TTS on H100 improved +48.5%/+76.3%, and on H200 +65.2%/+149.5%, but the H200 streaming baseline was relatively low, and the blog cautions readers to interpret this +149.5% carefully. Qwen3-TTS +25.2%/+4.7%, Qwen3-ASR +14.2%/+5.0%, Whisper +7.6%/+12.2%, MOSS-TD +18.8%/+30.0%. Ming3-TTS’s 0.5B model on H100 improved +326.5%/+93.9%, and on H200 +344.3%/+114.5%. The Ming3 range includes 28 other PRs; #3136 also fixed a sampling bug in the baseline, and the benchmark reuses reference recordings, so not all of this can be credited to OpenRSI. The blog lays out these caveats, which is more credible than simply presenting the biggest numbers.

On speed: in the best search runs for the first six models, five produced their first candidate passing all guards within 8 hours. Over 24 to 48 hours, gains compounded to 1.44× to 2.20×. Note that each task has its own benchmark; reward values in the figure cannot be compared across panels and cannot be read directly alongside the throughput gains in Table 1. The guards check latency and WER, and candidates that are fast but degrade quality do not receive reward.

There is an unanswered question here. A commenter asked in the replies: if an acceleration passes all tests but changes behavior that WER cannot catch, who is responsible for finding it? Will the harness roll back already merged PRs? The blog does not answer this directly. It only says output correctness and quality are checked within the coverage of the reported tests, and it does not guarantee that waveform or perceptual quality remains unchanged. The article itself acknowledges that WER is not sufficient for speech. This is the most practical hurdle once RSI lands in production infra: the evaluator is the product; if the guards are poorly written, the agent will optimize what you measure.

OpenRSI says the next step is to hand more of the loops still run by humans today over to agents, moving toward fully autonomous RSI for AI infrastructure. It sounds like a conference keynote title. But the checkable part is already on the table: 14 merged PRs, public tasks, harness, and blog, reviewed by maintainers, with data that includes caveats. RSI will prove itself through PRs like these: reproducible, reviewable and rejectable, and mergeable. There is a fireside chat about this at SGLang Summit 2026, on November 12–13, at Fort Mason in San Francisco.