Tsinghua Open-Sources AKV: Directly Optimizes KV Cache, Reducing Average Agent Cost by 27.7%

On October 26, 2026, Asif Razzaq, author of MarkTechPost, reported that Tsinghua researchers have open-sourced AKV. This is a service system based on SGLang—an open-source LLM inference service framework—and AKV is mainly used for KV Cache engineering in Agent scenarios.

The longer an Agent runs, the longer its history becomes. User messages, tool calls, external responses, and intermediate inferences are all accumulated in the context. A common mitigation method is summarization, which compresses old history into short texts before feeding them to the model. Summarization reduces tokens but may lose details and also generate an extra model call.

KV Cache can be understood as the cache left during model inference. For each token processed, the model generates corresponding Keys and Values, and subsequent rounds directly read these results instead of recalculating old content. The problem is that caching takes up video memory, and the longer the history, the greater the scheduling and cost pressure.

AKV’s approach is to move the optimization position from the text layer down to the cache layer. The system directly discards and relocates the KV Cache: deletes the cache not needed for current execution, and moves useful cache to appropriate positions. Agents do not need to rely on summarization to maintain all history; what is retained is the original computational state.

According to the report, AKV reduces the average cost of four Agent benchmarks by 27.7%. The report does not specify the specific statistical standard for cost; this figure should be regarded as a research benchmark result, not representing that all online businesses can achieve the same reduction.

What is notable about this is that Agent cost optimization is no longer limited to prompt engineering and context trimming, but has begun to delve into the internal cache lifecycle of inference services. For ordinary users, it can be simply understood as enabling AI to reduce repeated computational costs for old conversations.

Source: MarkTechPost

In your Agent business, what is the most costly: old tool returns, long documents, or cache left from multi-round inference?