65% and 34%: Decagon Lets Agents Tune the Inference Stack in Production Traffic Replay

Decagon has published a system called AutoResearch. It breaks inference optimization into diagnosis, experimentation, and measurement, then has agents execute these steps in a loop. SGLang's official repost notes that the system has already been used in Decagon's inference stack. The original authors are Pavi Dhiman and Cyrus Asgari; the full article is here, and SGLang's repost is here.

Decagon serves more than a dozen model endpoints online. Each endpoint has different architecture, load, latency requirements, and hardware configuration, and optimizations can interact with one another. Trying them one by one manually keeps getting more expensive. Inference also happens to be a verifiable task: replay the same load, measure the latency-throughput curve, and check whether successful requests per second per GPU improve within the SLA, and agents get clear feedback.

The AutoResearch loop is not complicated: first measure the current deployment, diagnose bottlenecks, propose changes, and run experiments on GPUs. The search space includes serving parameters, GPU layout, speculative decoding, and inference engine patches. After finding an improvement, it first validates it, then turns the result into a new baseline and continues looking for the next bottleneck. The system runs on Devin Cloud, with Modal providing the environment for GPU training, profiling, and replay.

Two design principles underpin the system: replay must be faithful enough to production, and the research loop should converge as much as possible to a global optimum. Replay requests are reconstructed from production logs and first go through PII redaction, while preserving message structure, system prompt, tool definitions, output schema, and generation parameters. Redaction changes token lengths and shared prefixes, so the system also checks whether replay still represents production.

Replay comes in two kinds. A capacity test sends requests at increasing arrival rates, holds each rate for several minutes, and randomizes request intervals, plotting a latency-throughput-load curve to find the highest test rate that can be sustained within latency and error limits. A production-shape test converts production request counts or timestamps into per-interval arrival rates, reproducing bursts, sustained busy periods, and idle periods, and also checks whether autoscaling adds capacity fast enough during bursts and releases GPUs when demand drops.

The executor is written in Rust and sends requests using open-loop scheduling, without waiting for the previous response to finish. When the server slows down, traffic keeps arriving, exposing queueing pressure. Baseline replay must match production's token-length distribution, fraction of cached input tokens, request timing, and concurrency. Matching only the average request rate is not enough: if lengths, prefix reuse, or burstiness change, bottlenecks change, and optimization gains may be overestimated.

Diagnosis relies on logs, production metrics, serving configuration, and GPU profiling. Profiling looks at both the time breakdown, to identify expensive operations such as attention or MoE computation, and the execution timeline, to determine whether prefill interrupts decode, whether CPU work overlaps with GPU work, and whether kernel launch gaps leave the GPU idle. Diagnostic workloads change only one traffic property at a time, such as changing prompt length while fixing output length, comparing shared prefixes with unique prefixes, or inserting long prompts during decode. Tests that have been run are distilled into recipes and reused for endpoints with similar performance characteristics.

Orchestration consists of one orchestrator agent and multiple subagents. The parent builds replay pools, runs diagnostics, then splits work into N directions and opens N subagent sessions. Each subagent receives a campaign ID, experiment ID, replay pool ID, latency requirements, baseline measurements, historical findings, and a research question. It reads source code, documentation, profiles, and prior experiments, and returns four parts: an explanation of what limits performance, which configuration or engine patch the intervention should change, a prediction of how metrics will change, and what result would falsify the hypothesis.

Only the orchestrator runs formal experiments. It centrally manages deployment, warm-up, traffic generation, measurement, and scoring, and schedules concurrent experiments within GPU capacity. Results are classified as lossless or lossy. Changes such as scheduling and memory management that do not alter model outputs are considered lossless; weight or KV cache quantization may affect quality and is considered lossy. Promising candidates must be tested repeatedly, and then a human reviews the performance-quality trade-off to decide whether to merge them.

Once an improvement is confirmed, it becomes the new working baseline, while the original baseline continues to serve as the reference for total gains across the whole campaign. Agents re-diagnose at the new serving limit. Two independently measured optimizations may compete for the same resource, so combined candidates need controlled comparisons and ablations to confirm which one actually contributed the result. If a direction repeatedly produces no measurable gain, the orchestrator changes the hypothesis. A campaign ends at the deadline or when there are no more evidence-supported experiments, and the best configuration and open questions are saved.

Speculative decoding is a separate stage. The goal is to adapt the drafter to the fine-tuned target model and specialize it for production load. Optional starting points include MTP heads, DFlash, DFlash2, EAGLE, and DSpark. On a Gemma endpoint, Decagon found that the served acceptance length of an existing DFlash2 drafter was about 17% higher than that of the production MTP head, and continued training raised it another 14% above this starting drafter. Training data covers tasks, prompt and output lengths, languages, conversation stages, tool use, and structured output; hard examples are weighted by how well the drafter predicts the target output; rollouts use positions where the target rejected the drafter as supervision; and sampling density and weighting methods such as D-PACE are also judged through experiments.

Guardrails include limits on concurrent GPUs, time, and GPU-hours; a limit on the number of training variants; new code must first pass CPU checks and a 50-step GPU rehearsal; replay load, latency limits, and correctness requirements are fixed; and the serving pool does not participate in training or checkpoint selection. Confirmed gains generate deployment PRs, which go to production only after a human reviews the quality trade-offs.

Shared memory is stored in version-controlled files. Per-endpoint memory avoids rerunning changes that have already failed or been neutral; cross-endpoint memory records bottlenecks, techniques, models, hardware, load, and engine versions, and surfaces findings from similar constraints. It also preserves disagreements: speculative decoding helps on some endpoints and hurts on others, and both results are kept so that the next round can find the conditions that explain the difference.

In terms of results, AutoResearch has found gains in quantization, kernel selection, scheduling, GPU execution, topology, and speculative decoding. In the Gemma deployment, tuning the prefill-decode interval increased replay capacity by up to 65%; adapting a DFlash2 drafter for the select-playbook endpoint increased throughput by 34% relative to the production drafter. The original article has a results table listing requests/s/GPU gains according to each endpoint's production latency requirements, with baselines being SGLang defaults or Decagon's current tuned configuration.

The original article includes load curves, production-shape replay, GPU profiling, sub-agent orchestration, the full loop, and the results table, and it is worth reading alongside this piece: AutoResearch full article. In the replies, readers also asked: why not let subagents run experiments themselves, and is there any back-and-forth between subagents and experiment results; how much of the replay noise comes from LLM output uncertainty, and can SGLang's deterministic kernels help. The main text does not directly answer these two questions, but they are details worth watching as the system evolves.