Cache hit rate rises from 30% to 95%, but TTFT still needs to be broken down by cache tier
SGLang HiCache brings long-prefix reuse to context parallelism, pipeline parallelism, speculative decoding, and PD-disaggregated serving. After iFlytek Astron MaaS integrated external storage, the cache hit rate for multi-turn agentic workloads rose from 30% to 95%, with median TTFT below 3 seconds. However, an external-storage hit is not the same as a GPU memory hit; TTFT also needs to be viewed by tier.
Agentic workloads have a very concrete waste: multi-turn calls repeatedly consume the same long prefix. LMSYS Org describes how SGLang HiCache works in parallel and distributed serving, with the goal of making this prefix reuse truly span context parallelism, pipeline parallelism, speculative decoding, and PD-disaggregated serving. It also mentions a unified KV layout so caches can be reused across compatible parallel configurations.
Production data comes from the iFlytek Astron MaaS team. Its agentic CodingPlan workload is served by GLM-5.1/5.2, using PP, PD Disaggregation, and MTP. After integrating HiCache with external storage, the cache hit rate for multi-turn agentic workloads rose from 30% to 95%. These inputs are usually 30K to 200K tokens, up to 1M. Median TTFT is below 3 seconds; previously, latency had made the service basically unusable.
The numbers are impressive, but one reply is worth viewing in the same picture: an external-storage hit is not the same thing as a GPU memory hit. Different cache tiers mean different TTFT composition. A 95% hit rate shows that reuse is working; next, it is more important to look at median TTFT broken down by cache tier. Otherwise, an overall hit rate can easily obscure the cost of backfilling from external storage to the GPU.
Blog post: http://lmsys.org/blog/2026-11-02-hicache-distributed-serving
Original post: https://x.com/lmsysorg/status/2117347410716561767
发布时间: 2026-11-03 22:40