DeepSeek puts KV cache on SSD, making first token 26x faster—provided not a single character in the prompt is changed

DeepSeek has open-sourced another inference stack, DeepInfer. It addresses one of the most easily overlooked waits in large-model serving: when a request repeats a long prefix—such as system prompts, tool definitions, or previous conversation turns—can the server directly reuse the already-computed KV cache instead of running prefill again?

The official numbers are straightforward. After a 128K-token prompt hits the cache, first-token latency drops from 13 seconds to 500ms. On one day of production traffic last year, 56% of input tokens were read directly from disk.

Hard drives are usually the unwanted player on the request path. Only when GPU memory is not fast enough or not large enough would one consider putting KV cache elsewhere. DeepSeek deliberately puts the cache on SSD and makes it work in the main flow.

Why disk is usually avoided

Before an LLM generates, it must first read the entire prompt and compute keys and values for every layer and every token. This is prefill. With long prompts, prefill is the slowest part.

Many requests start exactly the same way. Chat resends history turn by turn, and agents carry the same set of system prompts and tool instructions on every call. If these KVs are stored, the next call only needs to compute the new tokens.

The problem is that KV cache is huge. On standard models, the KV for a long prompt can reach tens of GB. Reading that much data from disk may be slower than recomputing on GPU. So inference engines generally keep KV in GPU memory, which is small, old prefixes are quickly evicted, and repeated requests often miss.

DeepSeek shrinks the cache, then spreads reads wide

DeepSeek-V2 introduced MLA, Multi-head Latent Attention. Instead of storing independent KV for each attention head, it stores one compressed latent vector per token per layer, and reconstructs keys and values during attention computation. In the V2 paper, this step reduced KV cache by 93.3% relative to the earlier 67B model.

Only when the cache is small does it become eligible for disk. Fewer bytes need to be read per hit token, so loading from SSD can be faster than recomputing.

A single SSD is still not enough. The cache is placed on DeepSeek's self-developed distributed file system, 3FS. 3FS splits files into chunks and distributes them across thousands of SSDs on multiple storage nodes. Reads are pulled in parallel, and bandwidth adds up. When a request arrives, the system matches cache blocks by prefix, reads from 3FS into host memory, then copies layer by layer to the GPU while the model has already started running. Every prefill machine can access all cached prefixes, so a request does not need to return to the machine that served it last time.

Avi Chawla mentioned in a tweet that cached prefixes are retained for 3 days for agent traffic and 7 days for chat. On deepseek-flash, the price of a cache hit is 50x lower than a miss. A user asked whether the SSD cache survives restarts. Avi replied yes: KV blocks are placed in an independent 3FS storage cluster, separate from GPU machines, so engine restarts or redeployments do not affect it.

Hit conditions are stricter than you might think

This mechanism only recognizes exact matches starting from the first token. Stable content such as system prompts, tool definitions, and documents must be placed at the very front, while variables such as timestamps, request IDs, and usernames go at the end. Someone merely added a timestamp to the system prompt, and all subsequent blocks were invalidated. Changing the order of tool schemas also voids the entire cache.

Another easy pitfall is history summarization. Summarization rewrites the prefix, and the next call is billed at full price for cold tokens. Truncating tool output and keeping prefix bytes identical are required for the cache to survive. To confirm where two prompts actually diverge, comparing token IDs is more reliable than reading log text. BOS tokens, trailing newlines, and reserialized tool schemas can all change matching in ways invisible to the eye.

Avi Chawla's long article breaks caching in LLM serving into four layers that are easy to confuse:

KV cache: stores attention tensors within a single request and releases them when the request ends.
Prefix cache: keeps KV blocks on the server after a request ends, indexed by a hash chain of token IDs, so later requests can look them up.
Prompt cache: cloud providers turn prefix reuse into a billing item, charging a premium for writes and discounting reads.
Semantic cache: converts requests into embeddings, performs similarity search, and returns a stored answer directly on a hit.

The first three are exact matching: outputs are unchanged, and only computation is saved. Semantic cache is fuzzy matching: a hit may give you a wrong answer, returned with a 200 status code. A user said in a reply that the biggest problem with semantic cache is timeliness. Ask the same question four hours apart and the correct answer has already changed, but embeddings cannot tell the difference. Another said tool calls with state queries cannot use semantic cache. Artem Tarasov's reply was more direct: the first three are transparent, semantic cache is the only one that may give a different answer, and lumping them together makes it easy to discover at launch that there was no evaluation.

Can this approach be ported to ordinary models?

A user asked whether this can be ported to ordinary MHA models, or whether it only works because MLA first shrinks the cache. The key lies in two conditions: the KV cache must be small enough, and storage must be wide enough. MLA solves the former, 3FS solves the latter. If an ordinary model still has a very large KV cache, too many bytes must be read per hit, and disk latency becomes the bottleneck again.

With this open-sourcing of DeepInfer, DeepSeek lays out the cache path in its inference stack for all to see. It does not change model capabilities; it simply saves the money and time of repeated computation. For agents and long system-prompt scenarios, this saving is very practical. For people who change the beginning of their prompts every day, the 26x speedup basically remains in papers and tweets.

Reference sources: Avi Chawla's tweet and long article https://x.com/_avichawla/status/2115400590960599112 , and the LMCache project https://github.com/LMCache/LMCache .