Wink Pings

Your LLM Isn’t Writing an Answer—It’s Running the Model 500 Times

A 500-token answer means the model runs 500 times. Which has more impact on the user experience, TTFT or ITL? Why is the KV cache bigger than the model?

Your LLM isn’t writing an answer—it’s running the model. A 500-token response means the entire model is executed 500 times, each time predicting only the next token.

Most people are forced to understand LLM inference only when something breaks. The model runs fine on a laptop, but once it’s behind an API, twenty users come online, and GPU memory explodes. Or latency goes from smooth to painful, and nobody can explain why. These problems basically come down to a few foundational concepts: how the model generates text, what the KV cache is, and how they occupy GPU memory.

Start with performance metrics. A production LLM service usually has two kinds of SLOs: accuracy and performance. On accuracy, optimizations like quantization slightly change model behavior and need to be validated against the benchmarks on the model card. For example, with the FP8-quantized version of Llama 3.1 70B, weight memory is halved while average accuracy remains at 99.88% of the original; an individual benchmark exceeding 100% is just noise.

On performance, you need to distinguish several speeds. TTFT (time to first token) determines the user-perceived “response speed,” while ITL (inter-token latency) determines whether the text flows smoothly. Rough formula: request latency ≈ TTFT + ITL × (number of output tokens - 1). For a 200-token answer with TTFT 300ms and ITL 30ms, total latency is about 6.3 seconds. At that point, halving ITL saves 3 seconds, while halving TTFT saves only 150ms. So the longer the response, the more important ITL is. Throughput and latency also pull against each other: packing more users onto a GPU raises total throughput, but usually worsens ITL for each individual user. Inference engineering is largely about choosing a position on this curve.

Next, look at the internal mechanics. In the self-attention of each Transformer layer, the current token uses three projections—query, key, and value—to look things up in a “library”: query is the retrieval request, key is the spine label, and value is the book’s contents. To compute the output for the current token, you need the keys and values of all previous tokens; the current token’s query is used only for itself. Hence the KV cache: after computing the K and V for each token, you store them and read them directly later. Without it, every new token would require recomputing the K/V of all previous tokens, and the work would grow with the square of the length.

The KV cache is a cost you have to pay. Formula: KV memory per token = 2 × number of layers × number of KV heads × head_dim × number of bytes. Llama 3 70B: 80 layers, 8 KV heads, 128 dimensions, BF16, about 320KB per token. It looks small, but with long context it becomes frightening: at 128K context, the KV cache for a single request reaches 40GB. Model weights are only 140GB—seemingly large, but weights are fixed and shared by everyone; the KV cache is independent per request and grows with tokens. With 10 long-context users, total KV cache exceeds 400GB, which a single 80GB H100 simply cannot hold.

This also explains why Llama 3 70B uses Grouped-Query Attention (GQA): 8 query heads share one set of K/V. If every query head had its own K/V, the KV cache would balloon 8x, and a single request at 128K context would be 320GB. GQA is not a minor optimization; it is a prerequisite for long-context models to be commercially served.

Once these foundational concepts click, optimizations such as batching, quantization, and PagedAttention look like natural extensions. Sid’s series of posts on X explains these two parts very clearly; the data comes from the original articles: Part 1: How Inference Works and How to Measure It and Part 2: Self-Attention and the KV Cache.

发布时间: 2026-11-22 19:59