Why do long contexts use more VRAM? The recomputation KV Cache saves and the attention it doesn't

KV Cache solves a very specific problem: during autoregressive generation, every time the model emits a token, it must use the current token's Q to attend to the K/V of historical tokens. If the historical tokens' K/V were recomputed at every step, the cost would grow with context length. KV Cache stores these K/Vs and reuses them directly in the next step.

The generation process can be split into two phases.

In the prefill phase, the model reads the prompt, processes the entire input at once, and computes and caches each layer's K/V. In the decode phase, each step processes only the new token, computes its Q, K, V, appends the new K/V to the cache, and retrieves historical K/V from the cache. The tweet author ramakrishna e/acc specifically emphasizes one point: KV Cache does not eliminate attention to historical tokens; it eliminates recomputation of historical K/V. This distinction is crucial; otherwise it is easy to misunderstand inference optimization.

The longer the context, the higher the KV cache footprint, and it can become the main bottleneck. It is affected by the number of layers, number of KV heads, head dimension, sequence length, and batch size. Hence MQA and GQA reduce the number of KV heads, PagedAttention manages memory in pages, and Prefix Caching reuses identical prefixes. These techniques all address pressure in the same direction: how the cache can use less VRAM, move less data, and reuse more.

The original tweet includes an explanatory video:

For people building inference services, KV Cache directly affects throughput, latency, and VRAM budget. For ordinary users, it explains why long conversations consume more resources and why identical long prefixes can be reused. Understanding KV Cache is necessary to see what those optimizations in LLM inference systems are actually saving.

Source: ramakrishna e/acc's tweet, original link.