Cache hit, yet tokens are still being recomputed: FlashState triples vLLM's agent throughput

FlashState is now open source. It is an inference system that manages prefix caching for hybrid LLMs, with a very clear target: agentic workloads. The authors are from UIUC, UT Austin, and Together AI, and the code, paper, and project page have all been released.

Code / Paper / Project page

First, look at what problem it solves. A coding agent resends an ever-growing context every turn. In a standard attention model, caching the KV lets you reuse what was computed earlier. A hybrid model mixes attention layers with linear attention or state-space layers, and restoring a prefix requires two things: the attention KV for every token, and a recurrent checkpoint at the exact recovery position. Only when both are present is it a hybrid state.

vLLM stores recurrent checkpoints only at KV-cache block boundaries, and KV blocks are evicted by recency of use. The end point of the previous turn often falls in the middle of a block, so the next turn can only resume from the previous boundary, and the tail tokens are recomputed. Under high load, the entire context may also be evicted. The result is that the cache appears to hit, but the recovery point is not there, so the prefill work is wasted.

FlashState's approach has three layers:

• Arbitrary State Checkpointing, ASC. Recurrent checkpoints are no longer tied to KV block boundaries and can be stored at any token position. A complete hybrid state is captured when a request ends; if the target position is inside a long prefill chunk, the kernel exports the checkpoint as a side output, and the chunk continues computing.
• Cost-aware hybrid state management. A checkpoint and the KV it depends on are treated as one unit and priced by how much prefill time each block of memory can save; admission and eviction share the same priority scheme. No workload-specific tuning is required.
• Optional Mooncake store. The hybrid state of each completed request is written to a Mooncake store in host memory, and the master evicts entire states according to the same priority.

The numbers are quite direct. In the GitHub figure caption, after replaying 48 coding agents on the same batch of GPUs for 30 minutes, FlashState completes 20 agents, at 61.7K output tokens/min and an 87% cache hit rate; vLLM completes 7, at 22.0K and 30%; SGLang completes 8, at 15.4K and 6%.

On GLM-5.3-Flash, NVFP4, 1M context, two B200 GPUs per engine, and 48 concurrency, prefill throughput is 4.66M tokens/min/GPU, compared with vLLM's 1.48M, about 3.14x. Median uncached tokens drop from 24.7K to 1.03K, p90 TTFT drops from 20.4 seconds to 2.36 seconds, an 88% reduction. Decode throughput is 3.02x.

There are also results for Kimi-K3.1-Pro on GB300: one engine with two GB300 trays, 8 GPUs, decode context parallelism 8, DSpark speculative decoding, and a 640 GiB Mooncake store. Peak output throughput per GPU is 8,683 tokens/min, compared with vLLM's 6,377, 36% higher; p90 TTFT is 2.0 seconds versus 9.1 seconds.

In the Mooncake setup, with 512 GiB of host memory used as cache, FlashState raises peak input throughput by 26% and reduces p90 TTFT by up to 73% at the same concurrency. In a reply, the authors say this approach is orthogonal to disk offload. Disk or S3 can expand cache capacity, but exact recovery points still depend on ASC, and scenarios with long waits for tool calls or sub-agents still depend on dependency-aware eviction.

The tweet's framing is more concise: on GLM 5.3 Flash, FlashState is more than 2x ahead of vLLM in both throughput and provider profit, with 3.7x lower latency and 2.5x lower user cost. The demo numbers on the project page are output tokens/min from 22.0K to 61.7K, P50 end-to-end latency from 46.1 seconds to 12.4 seconds, and P50 TTFT from 3.30 seconds to 0.20 seconds.

There are also caveats to keep in mind. The 3.14x is prefill throughput, for one model, one workload, and one hardware setup; it does not mean universal end-to-end speedup, nor does it mean fewer generated tokens. In the paper, in-chunk export was only tested on GLM's KDA kernel; Qwen and Nemotron use capture at request end. FlashState is still research code, and its options will change.

This kind of work does not change model capabilities, but it does change the unit economics of agent products. Model leaderboards change every day; behind the scenes, cache recovery points, eviction policies, and host offload ultimately show up in the bill and in first-token latency. People building agent infra should reproduce it once; for those who only watch model leaderboards, at least remember one thing: a cache hit does not mean the recovery point is still there.