At 3 a.m., the inference system begins modifying its own engine
Inferact wants to build an autonomous inference system around vLLM: it finds bottlenecks on its own, writes kernels, runs benchmarks, and merges the winners back into vLLM. Simon Mo shared two production cases and also admitted that it is still far from fully automated.
Woosuk Kwon reposted a long article by Simon Mo explaining what Inferact is actually doing and how it relates to vLLM. The questions are concrete: what stays in the open-source project, who maintains it, and what Inferact gets out of it. Simon Mo's answer is direct: Inferact builds an autonomous inference system around vLLM; vLLM remains open source; and every engine optimization found by the system and the team flows back into vLLM. The original article is here, and the repost link is here.
First, a bit of context. vLLM is an open-source inference engine. When a model goes into production service, speed and cost depend heavily on this layer. Simon Mo narrows the problem sharply: inference is the only problem they are trying to solve.
The trouble with inference is combinatorial explosion. Every week, labs release new model architectures; chip vendors design hardware without ready-made tuned kernels; and customer traffic shifts from short chats to long-context agents. Each new combination must be tuned separately to run stably and fast. Models, hardware, and workloads each evolve independently, and the gap between them eats into throughput and reliability. vLLM's job is to bridge that gap, but the search space includes more than 5,000 models, over 1,000 chip architectures, and nearly unlimited workload shapes. Exhaustively searching it by hand is unrealistic.
Inferact's approach is an autonomous inference system, AIS. Autonomous means: give it a goal, and it advances on its own, finds its own feedback, and does not need a human to tell it what to do next. According to Simon Mo, the system observes production workloads it runs or is authorized to observe, learns from community sharing, and does not need access to customers' deployments; it adapts to new models, new hardware, and changes in customer traffic; it evolves its own code, as well as the “brain” that accumulates knowledge and engineering judgment; it gets around roadblocks like a strong engineering team; and it continuously optimizes against each workload's own goals. Humans set the goals and provide quality standards and ideas.
What does this system look like in action? Simon Mo gives an example from an ordinary day.
At 3 a.m., a lab releases a preview of a new open model. Before Inferact employees wake up, the system starts profiling. The new attention variant has no tuned kernel and can only fall back to a generic path, missing the latency target.
At 4 a.m., the system starts Hillclimb on its own: some agents write a megakernel, others fuse the new operator into existing kernels, and another rewrites the engine from scratch.
Overnight, several candidates win in isolation but lose end to end. Only combinations that pass a full serving benchmark keep moving forward.
In the morning, that combination is deployed according to customer workload requirements and chip architecture, and the winning changes are merged back into vLLM.
That evening, the system reviews its day and finds that most failed candidates died at the same end-to-end gate, so it rewrites the agents to run that check first.
The next day, it continues.
This sounds like science fiction, but Simon Mo gave two examples that are already running in production.
One is MiniMax M3.1 Flash. It brings a new architecture: sparse attention with an indexer, large MoE, visual input, and native DSpark speculative decoding. vLLM needs to run production service with optimized performance on NVIDIA and AMD on day 0. The trouble is that the workload keeps changing, from long-context agentic coding to short-input long-output, and then to decode-bound service after the final checkpoint doubles output length. Their approach is to use the same branch for open source and production, shadow 100% of online M3 traffic to M3.1 before shifting traffic, then run production Hillclimb against this traffic, promoting changes from local gates to online shadow eval. The results include: p50 time to first token dropped from 1.5 seconds on the first non-PD endpoint to 0.8 seconds in production; a patch to the expert-parallel prefill path reduced TTFT by 60% to 70%; pipeline-parallel prefill had 19% to 46% higher throughput than tensor parallel; and AMD output per card improved by 56%. Missing kernels, including MSA attention, indexer, and NVFP4 MoE, all went upstream, and day-0 images were released for NVIDIA and AMD.
The other is Agent V on Kimi K3.1. Agent V is their shared agent infrastructure. Ask it in Slack, and it will complete tasks in a sandbox, with native support for Hillclimb. When connecting Kimi K3.1 to their own serving stack, output quality dropped for certain batches. A 16-agent Hillclimb reproduced the issue; the root cause was traced to an attention routing switch that depended on batch composition, and a fix was deployed afterward. The fix raised the DeepSWE score from 0.558 to 0.602, matching the model's reference accuracy. On the prefill side, a 24-hour Hillclimb made TTFT for 800k context on GB300 2.78× faster. On the decode side, a 30-hour Hillclimb reduced p99 per-output-token time by 35%. K3 production throughput on GB300, from mid-September to the end of October, rose from about 0.77M tokens/min per GPU to 1.4M tokens/min, roughly 1.8×. Agent V mostly runs on Kimi K3, served by vLLM; about 79% of natural sessions and 53% of tokens come from their own open-weight deployments.
But Simon Mo did not overstate it. AIS is the north star, and it is not here yet. Compute is scarce now, and they cannot simulate the world well enough to test traffic that has never happened; the feedback loop is also still narrow, coming only from deployments they run and information shared by customers and the community. What comes next is one-click observability so the system can see more; Agent V across partners and the vLLM community so the whole ecosystem gives it feedback; and letting the system observe workloads and adapt to models and chips, improving itself without waiting for them to step in.
In this design, vLLM is not only the object being optimized; it is also the entry point through which the system obtains real requirements. The open-source project provides composability and community knowledge, while Inferact handles automated optimization and the production pipeline. The risk lies here too: the more the system depends on traffic it runs itself, the more easily it mistakes known workloads for the whole world; and when compute is insufficient, the pace of self-improvement gets bottlenecked by benchmark-running capacity. Simon Mo writes these limitations clearly and admits there will be many failures.
What is really worth watching is whether that 3 a.m. scene can become routine: the system finds the gap itself, writes candidates itself, runs the full benchmark itself, and merges the winner back into the open-source project itself.
发布时间: 2026-11-19 17:37