LLM Speed Isn't Determined by Model Weights

Amit Shekhar posted a roundup on X, listing six LLM inference optimization techniques as a series of articles. The original post makes their purpose clear: KV Cache, Paged Attention, Flash Attention, GQA, Speculative Decoding, and Continuous Batching let LLMs run fast and handle concurrency in production environments.

Background first. LLM text generation emits one token at a time, and each token requires attention computation. Users wait for the first token and also care about the speed of subsequent tokens. The server side cares about how many people the same GPU can serve simultaneously. Model weights determine the capability ceiling; the inference stack determines cost and concurrency. Model leaderboards rarely score these engineering details, but the GPU bill does.

Amit Shekhar's series breaks the six techniques down one by one:

• KV Cache: Cache the Key and Value in attention, reuse historical computation results, and avoid starting from scratch every time a token is generated.
• Paged Attention: Solves the GPU memory waste of KV Cache by borrowing the operating system's approach to memory management, allowing the same card to serve more requests.
• Flash Attention: Standard attention is slow because of back-and-forth reads and writes between GPU high-bandwidth memory and on-chip SRAM. Flash Attention uses tiling, online softmax, and recomputation during the backward pass to reduce data movement; it is very common in modern LLMs.
• Grouped Query Attention: From MHA to MQA to GQA, it reduces the number of KV heads and lowers memory and bandwidth pressure; it is a compromise between MHA and MQA.
• Speculative Decoding: A small draft model guesses multiple tokens first, and a large target model verifies them all at once. The original post mentions that rejection sampling mathematically guarantees no drop in quality, with a practical speedup of 2x to 3x.
• Continuous Batching: Static batching must wait for the longest request to finish, so the GPU easily sits idle. Continuous batching dynamically adds new requests, keeping every generation step busy.

These techniques address three things: redundant computation, memory waste, and GPU idling. They do not change the model's knowledge; they change how many people can be served per unit time and the cost per token. When launching a chatbot, users experience speed, while bosses see the bill. Neither side can avoid these details.

The original post writes each of these up as a separate long article, with links above. Original post: https://x.com/amitiitbhu/status/2054100147546837154