The Same System Prompt Gets Recomputed 170x—The Problem Is the Routing Layer
In production, the beginning of each request’s prompt is almost identical; only the end is new. But a router that doesn’t look at the cache will send the next round of requests to another GPU, forcing it to recompute the entire prefix from scratch. In a test with 8 vLLM pods and 16 H100s, p90 time to first token dropped from 92.5 seconds to 0.54 seconds—and that gap comes from exactly this.
Production LLM requests are mostly repeated tokens.
When the same Agent is running, the beginning of every request in each round is the same system prompt, the same tool definitions, and the same conversation history; only the last part is new. The GPU that served you in the previous round still has the intermediate state for those prefixes in its KV cache.
The problem is that if the router doesn’t look at the cache, the next round of requests may be thrown onto another GPU. That GPU doesn’t have this prefix and must prefill it from scratch before it can emit the first token.
Avi Chawla breaks this process down clearly: prefix-aware routing is exactly what solves this.
It works in several steps:
A request comes in with prefix blocks A through D and a new suffix E. The router hashes the prompt by block, with each hash covering everything before it.
These hashes look up a block-to-worker index. This index stores only locations, not the KV tensors themselves.
Suppose Worker A matches 2 blocks, Worker C matches 3 blocks, and Worker B matches all 4 blocks. The request goes to B.
B reuses the cached KV for A through D, computes only the KV for E, and then starts decoding. Once E is cached, a cache event updates the router’s index, and the next round of requests continues to land on B.
If no worker matches, the request does a full prefill on the worker selected by the router.
In a benchmark with 8 vLLM pods and 16 H100s, prefix-aware routing brings p90 time to first token down to 0.54 seconds. With the same pods, random routing is 92.5 seconds. About 170x.
A real router doesn’t look only at cache hits. In replies, Avi mentions that Dynamo’s KV router scores two things at the same time: the cost of the prefill still required after cache savings, and the current worker’s decode load. So a GPU with a full prefix hit but a long queue can lose to an idle GPU that needs to recompute part of the prefix.
One commenter put the production trade-off more bluntly: pure cache affinity creates hot pods—one worker gets crushed while the others sit idle. In practice, you compute a weighted cost at the gateway layer: (cache_hit_blocks * prefill_cost_saved) - (queue_depth * queue_latency_penalty).
The place this mechanism most easily breaks is when the prefix itself is invalidated. Put a current timestamp or a random request ID at the start of the system prompt, and all downstream block hashes fail to match; the entire cache is invalidated. Multiple commenters pointed out this pitfall in replies: push dynamic content to the end of the prompt and keep the earlier parts byte-for-byte identical.
There’s one more distinction worth remembering. Artem Tarasov pointed out in the discussion that KV cache, prefix cache, and prompt cache are all transparent: same output, less computation. Semantic caching is different; it may return an answer you would not otherwise have gotten. When these things are lumped together, it’s easy to skip eval at launch.
Prefix caching itself has limits. GPU cache can be evicted or lost on restart. One commenter mentioned storing KV on disk, where it can be reloaded byte-for-byte in vLLM and SGLang.
At the end of the day, the model hasn’t changed, the compute hasn’t changed, yet time to first token differs by two orders of magnitude. The routing and caching layer is worth more than many people think.
发布时间: 2026-10-30 01:34