vLLM Begins to Modify Itself: Inferact Publishes Production Data for Its Autonomous Inference System
Simon Mo explains the boundary between Inferact and vLLM: vLLM remains open source, and Inferact builds around it an inference system that can tune itself and modify its own code, with optimizations fed back upstream. The article also provides production data for MiniMax M3.1 Flash and Kimi K3.1.
Simon Mo published a long post explaining the relationship between Inferact and vLLM. Many people have asked: what stays in the open-source project, who maintains it, and what exactly Inferact does. His answer is direct: vLLM will remain open source, and Inferact will build an autonomous inference system around vLLM. The engine optimizations found by the system and the team will all be fed back into vLLM for the entire community to use.
Original tweet: https://x.com/TheZachMueller/status/2123144022839492769
Simon Mo's long post: https://x.com/i/article/2123131336982712320
First, some context. vLLM is an open-source inference engine that handles serving after a model is deployed. Inferact is a team built around vLLM; according to the article, it brings together a group of inference engineers, many of whom have contributed to vLLM.
The trouble with inference is that three things evolve independently: model architectures change every week, the hardware provided by chip vendors may lack tuned kernels, and customer traffic shifts from short conversations to long-context agents. Every new combination has to be tuned separately, and poor tuning costs throughput and stability. vLLM has to bridge this gap. But Simon Mo says you cannot rely on throwing people at a brute-force search over a combinatorially explosive space: more than 5,000 models, more than 1,000 chip architectures, and no upper bound on the number of workloads.
Inferact's answer is an autonomous inference system built around vLLM. 'Autonomous' here has a very concrete meaning: given a goal, the system advances on its own, finds its own feedback, and does not need a human to tell it step by step what to do next. The article lists five things: observe production traffic or traffic it is authorized to use, and also learn from community-shared data; adapt to new models, new hardware, and changes in customer traffic; modify its own code as well as the knowledge and engineering judgment it accumulates; route around obstacles by itself; and continuously optimize against the goal for each workload.
This description sounds like automated machine learning, but the focus is on the dirty work of inference deployment: kernels, parallelism strategies, batching, routing, fallback paths, and end-to-end benchmarks. Simon Mo gives an example at 3 a.m. A lab releases a preview of a new model, and the system starts profiling while no one is awake. The new attention variant has no tuned kernel, so it first goes down a generic path, and latency misses the target. At 4 a.m. the system starts hillclimbing on its own: some agents write megakernels, some fuse the new op with existing kernels, and another rewrites the engine. Overnight, several candidates win in isolation but lose end to end. Only combinations that pass the full serving benchmark move forward. In the morning, that combination is rolled out according to customer workload and chip architecture, and the winning changes go upstream into vLLM. At night, the system reviews the day and finds that most failed candidates died at the same end-to-end gate, so it modifies its own agents to move that check earlier. The next day, it continues.
This process depends on production and open source feeding each other. On the production side, writing a standalone engine or making a single optimization is not hard, because inference tasks are usually verifiable. But isolated gains often are not robust enough once placed on the production distribution. vLLM runs production workloads and also has to scale through the distributed inference stack. To validate performance gains, only an engine tested by production traffic is trustworthy. On the open-source side, vLLM is where optimizations compound. The core engine is designed to be composable, so every release benefits all users. The autonomous inference system takes knowledge from the community, academic labs, and partners. Simon Mo says vLLM has the world's broadest knowledge base of inference requirements, and they will also have the system help triage issues and test pull requests faster.
Inferact has three things to do: keep vLLM leading in open source, with the team continuously optimizing and feeding changes back; build an autonomous system around vLLM, maintain the inference engine, and customize it by workload; and have humans set the goals. Simon Mo says both of these things require human taste to set the quality bar and provide ideas for improving the system. Inferact only wins if vLLM keeps getting better for everyone.
The article gives two examples already running in production.
MiniMax M3.1 Flash. M3.1 brings a new architecture: sparse attention with an indexer, large MoE, visual input, and native DSpark speculative decoding. vLLM has to deliver optimized performance from day 0 on NVIDIA and AMD, in production serving, while workloads are still changing: long-context agentic coding, short-input long-output, and eventually decode-bound serving. The approach is to use the same branch for open source and production, shadow 100% of live M3 traffic to M3.1 before switching, run production hillclimbing on that traffic, and promote from local gates to online shadow eval. Results: p50 TTFT dropped from 1.5 seconds on the first non-PD endpoint to 0.8 seconds in production; a patch to the expert-parallel prefill path reduced TTFT by 60% to 70%; pipeline-parallel prefill delivered 19% to 46% more throughput than tensor parallel; AMD improved output per card by 56%; missing kernels such as MSA attention, indexer, and NVFP4 MoE went upstream, and day-0 images for NVIDIA and AMD were released.
Agent V on Kimi K3.1. Agent V is shared agent infrastructure: ask it in Slack, and it completes tasks in a sandbox, with native support for hillclimbing. After Kimi K3.1 was deployed to its serving stack, output quality dropped for some batches. Agent V ran 16 agents to hillclimb, reproduce the issue, and root-cause it to an attention routing switch that depended on batch composition; a fix was rolled out. K3.1 quality: DeepSWE score went from 0.558 to 0.602, matching the model's reference accuracy. Prefill: a 24-hour hillclimb made 800k context on GB300 2.78x faster in TTFT. Decode: a 30-hour hillclimb reduced p99 time per output token by 35%. K3 in production on GB300: throughput per GPU went from about 0.77M tokens/min to 1.4M tokens/min, about 1.8x, between mid-September and late October. Most of Agent V runs on Kimi K3 itself, served by vLLM: about 79% of natural conversations and 53% of all tokens come from its own open-weight deployment.
Simon Mo also discussed the boundaries. AIS is the north star, but it is not here yet. Compute is tight right now, so it is impossible to simulate the world well enough to test traffic that has not happened yet; this is also the capability Agent V needs to safely rewrite itself. The feedback loop is still narrow, coming only from deployments it operates itself and from customer and community sharing. Next steps: build one-click observability for vLLM so the system can see more; build Agent V across partners and the vLLM community so the whole ecosystem feeds it; and build a system that can observe workloads, adapt to models and chips, and improve itself without waiting for them.
What is worth watching here is the way optimization happens. Today, inference engine performance depends on a small number of engineers who understand kernels, distributed systems, and hardware tuning by hand. If this approach works, the system proposes candidates, runs end-to-end tests, and rolls back on its own, while human work moves up a level: setting goals, setting the quality bar, and judging what is worth doing. Simon Mo is measured in his wording: AIS is not finished, but production data is already on the table. For vLLM users, the more practical short-term questions are: when will these upstream changes land in a release, and which kernels and parallelism strategies can be reproduced on their own workloads? For people building inference systems, Inferact has moved the competitive focus to automated hillclimbing and end-to-end validation, not just single-point kernel optimization.
发布时间: 2026-11-19 06:33