Why the KV cache stores only K and V: Q is discarded after use
Does an LLM have to recompute history every time it generates a token? The KV cache keeps K and V, while Q is computed once and thrown away. It saves compute—and brings memory pressure to the foreground.
The KV cache stores only K and V, not Q. The first time you see this design, it is easy to feel that something is missing. Q clearly participates in attention computation, so why not keep it?
Akshay Pachaar wrote a breakdown on X that explains this question from autoregressive generation all the way to the minimal dependencies of attention computation. The answer is not complicated; it is hidden in the inputs and outputs of attention at each layer.
An LLM generates one token at a time, and predicting the next token depends on all previous tokens. Without a cache, the model has to rerun the entire prompt at every step. The role of the KV cache is to skip repeated computation, and what it retains exactly reveals what attention truly reuses.
In each attention layer, every token produces three vectors. Q describes what the current token is looking for, K describes what index it provides to other tokens, and V carries the content it needs to pass along. Attention uses Q and K to compute scores, softmax turns the scores into weights, and then the weights are used to mix V.
The key is at the end. After one forward pass, each position has a hidden state. Only the hidden state of the last position goes into the logits, which are used to sample the next token. To obtain this hidden state, the last layer only needs the Q of the newest token plus the K and V of all historical tokens. The Q of a historical token serves only its own row; once that row is computed, it is finished, and later steps will not read it again.
Akshay uses a 4-token prompt generating the 5th token as an example. In the prefill stage, the model reads 4 tokens in parallel at once, and each layer computes Q, K, and V for each token. In the last layer, only Q4 of the last token is compared with K1 through K4; softmax turns the scores into weights, mixes V1 through V4, and produces one row of output. The FFN processes this position to obtain the final hidden state, and the 5th token is sampled from it.
After token 5 is added, each layer computes Q5, K5, and V5 for it, then compares Q5 with K1 through K5. K1 through K4 and V1 through V4 were already computed in the previous step. The causal mask ensures that a token cannot see later tokens, so adding token 5 does not change the historical K and V. Q4 serves only row 4, and that row is already complete and will not be used again in the future.
So the cache keeps all K and V. At each step, only one new Q is created, used once, and then discarded. One commenter put it vividly: the new token asks the question, and the old tokens are only responsible for answering; past queries are no longer useful after their own step ends.
This also explains why caching Q is not worthwhile. One commenter estimated based on Llama 3 8B that the KV cache is about 128 KiB per token, and about 1 GiB for an 8k context. Q has more heads, so storing it would only be larger, and later steps would not read it. Another commenter calculated that Qwen3-4B in fp16 uses about 147 KB per token, and about 4.8 GB for a 32k context, already exceeding the roughly 2 GB footprint of 4-bit weights. With long contexts and small devices, the cache often takes up more space than the weights.
The cost is right here. The KV cache trades memory for computation. Akshay's article mentions that Qwen 2.5 72B has 80 layers, a 32K context, and a hidden dim of 8192; the KV cache for a single request can consume several GB of VRAM, and with hundreds of concurrent requests, the cache often exceeds the model weights themselves. GQA and MQA reduce memory by sharing K and V heads, at little cost. Double the context, double the cache, and the number of users you can serve concurrently drops.
Slow first-token latency is also related to this. When a prompt comes in, prefill must process the entire input at once, computing and caching K and V for each token. After the cache is built, each subsequent token only requires a single-token forward pass, which is very fast. Akshay says KV caching brings roughly a 5x speedup in practice, and inference stacks such as vLLM, TGI, and TensorRT-LLM are all built on this idea.
The next time you see that there is no Q in the KV cache, there is no need to suspect that it forgot to store it. Q is a one-time vector, while K and V are the ledger that gets read repeatedly. If you are asked about this in an interview, do not just answer that Q is computed only once. Add the causal mask, the dependency on the final hidden state, and the memory overhead, and the answer will be complete. Akshay Pachaar's tweet is here.
发布时间: 2026-11-18 09:42