AI Optimizes Its Own Inference Stack, 14 PRs Merged: Behind the 300% Improvement, Humans Are Still Gatekeeping

14 upstream merged PRs, 8 speech and omni-modal models, and throughput gains of 7.6% to 344.3%. This is the first production-grade RSI assignment delivered by OpenRSI and SGLang-Omni. The original tweet is here: https://x.com/GenAI_is_real/status/2119910916762608057

RSI is often used to demonstrate benchmarks and toy problems. This time the goal is very concrete: make the omni-modal models in SGLang-Omni run faster, without changing model weights or slowing down other models. Agents propose hypotheses, design experiments, read results, and keep iterating; an independent verifier validates each gain, and the agent cannot see it. Each merged change becomes the baseline for the next round. Humans are responsible for defining tasks, benchmarks, guards, system design, and final review. The blog calls this path Harnessed RSI.

RSI-Anything turns a proposal into an executable benchmark, then lets agents iterate on code, reproduce gains, and open PRs. SGLang-Omni maintainers review over multiple rounds, sometimes writing code or reproducing results themselves, and finally decide whether to merge. The blog identifies five patterns across the 14 PRs:

• Idle threads should stop spinning and give CPU cores back to work in the queue. #2524 alone was measured as worth +32.5% streaming throughput.
• Don't convert tensors back and forth to Python lists across stages; keep them as tensors, and only carry typed bytes in the final message. #2555, #2778.
• Skip computation nobody uses, such as unnecessary quantizer reconstruction and duplicate reference audio encoding. #2525, #2882, #3071, #3136.
• Too many small GPU launches; use CUDA graphs or fused kernels to replay them in one go. #2779, #2859, #2880, #2901.
• prefill and decode on two machines should not take turns waiting; start the next request early. #2881, #3071, #3136.
  These changes almost all target runtime overhead around the model. The blog also documents the costs: #2859 adds about 41 seconds of startup time and 3.7 GiB of peak GPU memory; #2881 raised some streaming tail latency; and MOSS-TD's 64-client streaming first-text p95 also went up. The numbers depend on deployment configuration: the same #2524 is worth +82.9% on 8 cores, but only +33.6% on 16 cores.

On speed, the blog says that for five of the first six models, the first candidate to pass all guards appeared within 8 hours. OpenRSI's reply is more specific: in 4 to 8 hours it found speedups that passed the latency and WER guards, and in 24 to 48 hours those compounded to 1.44x to 2.20x. The main tweet also clarified the 300% figure: the model had only been supported for two days and was handed to OpenRSI without manual optimization, so the magnitude was exaggerated; most models that had already seen clear improvements were around 30%. The public summary gives a range of 7.6% to 344.3%; in the blog's table H100 tops out at 326.5%, while 344.3% appears in the H200 Ming3-TTS row; the blog also cautions that the H200 streaming baseline is low and that MOSS-TTS's +149.5% should be read carefully.

More worth watching is the shift in bottlenecks. Once the problem definition is clear, inference optimization shifts from 'limited expert attention' to 'limited compute, evaluator quality, and review bandwidth.' Agents optimize what is measured, so benchmarks, quality checks, and guards against reward hacking become the main engineering work. In speech scenarios, WER covers only transcription accuracy, not how the voice actually sounds. The blog itself acknowledges this.

A commenter followed up: if a speedup passes all tests but changes behavior in ways WER cannot catch, who catches it? Will the harness roll back merged PRs? The public materials do not provide an automatic rollback mechanism. The current defenses are still the independent verifier, latency and WER guards, and maintainer review. Median review time is about 3.5 days, with up to 23 commits, and human bandwidth remains a hard constraint. It is still far from fully automated RSI and looks more like an outer loop governed by a harness. 'AI is improving the infrastructure on which AI runs' is now something that can be put into practice; but the final gate is, for now, still in human hands.

RSI-Harness and RSI-Anything are now open source: https://github.com/OpenRSI-Foundation/OpenRSI-Index/tree/main/RSI-Harness . SGLang Summit 2026 will have related sessions on November 12–13 at Fort Mason in San Francisco: https://www.sglang.io/summit