Don't Look Only at the Compression Ratio for KV Compression: If Throughput Doesn't Improve, the Saved Bandwidth Will Be Paid Back

There are many papers on KV cache compression, and most first report the compression ratio: how many times it compresses, how much quality drops, and how much communication volume is saved. This metric is of course important, but in disaggregated LLM serving it is not enough.

First, some background. Disaggregated serving splits prefill and decoding across different nodes, and the KV cache needs to be transferred between nodes. The KV cache is the key-value state saved during model generation to avoid recomputation. The larger the transfer volume, the higher the network pressure, so compressing the KV cache looks like a direct and effective approach.

The problem is that compression itself also has to be executed. Distribution transformation, low-bit quantization, and lossless compression each have computation and memory access overhead. The paper's authors focus on the execution efficiency of the compression pipeline: modern serving systems are beginning to use high-bandwidth interconnects, and if the compression pipeline is not fast enough, a large part of the communication time saved will be eaten up by compression overhead. No matter how good the compression ratio looks, the end-to-end benefit may be wiped out.

This paper is titled Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving. The authors are Junhong Wu, Zedong Liu, Xinyang Ma, Yuhan Chen, Guangming Tan, and Dingwen Tao, and it was accepted by NPC 2026. The high-throughput KV cache compression data path proposed in the paper does two things. First, it fuses the distribution transformation and low-bit quantization stages into a KV-aware tilelang kernel, eliminating intermediate tensor materialization and redundant memory access. Second, it replaces the original lossless compression backend with an open, scalable, searchable backend based on LC, allowing a high-performance compression operator pipeline to run.

The results are direct. The fused kernel achieves up to 5.9x speedup; the new lossless compression backend raises compression throughput from under 90 GB/s to more than 500 GB/s; and on the evaluated workloads, task completion time is reduced by up to 2.04x. The paper's conclusion is that these optimizations allow KV compression to retain meaningful communication benefits under modern high-bandwidth networks.

The judgment worth noting here is that the evaluation criteria for KV compression schemes need to change. Looking only at the compression ratio makes it easy to ignore the throughput and end-to-end overhead of the compression pipeline. For people building inference services, when choosing a compression scheme they should look at at least two more numbers: compression throughput, and whether compression overhead under high-bandwidth networks has already canceled out the communication benefit. The compression ratio is part of the result, not the whole result.

Paper link: https://arxiv.org/abs/2611.04382

Original tweet: https://x.com/DPZ/status/2117957972835831811