LLM Inference Is Slow—Start with Prefill and Decode: Amit Shekhar's Learning Checklist

Amit Shekhar posted an LLM Inference Engineering checklist on X. It runs from Prefill vs Decode, KV Cache, and PagedAttention all the way to vLLM, SGLang, GPUs, and TPUs. The checklist itself is like a table of contents; the real value is the order: first understand the generation loop, then talk about memory, throughput, engines, and hardware.

Start by breaking down the generation loop

Amit Shekhar is the founder of Outcome School. He puts the starting point of the learning path on autoregressive generation and Prefill/Decode, which is a practical choice. Production issues often emerge here: why the first token is fast but subsequent tokens are slow; why a long context immediately consumes a lot of VRAM; why, as concurrency rises, the GPU looks busy but throughput does not go up.

Prefill reads the prompt and is compute-intensive; Decode writes the answer token by token and is more limited by memory bandwidth. A commenter in the replies mentioned that the distinction between Prefill and Decode explains why local Qwen feels normal for the first token but slows down afterward. This observation is not new, but it is accurate. Many inference optimization articles start with model architecture; this checklist starts with the generation loop, which is actually closer to the symptoms seen in production.

Memory and attention determine the cost of long sessions

KV Cache is the first key point. When the model generates each token, there is no need to recompute the historical Keys and Values; caching them can save a lot of repeated computation. Problems follow: the KV Cache for long sessions grows, VRAM is consumed, and the number of users that can be served simultaneously drops.

PagedAttention addresses the memory waste and fragmentation of KV Cache, borrowing the idea of paging from operating systems. Flash Attention handles the computational and memory-access efficiency of attention itself. Grouped Query Attention reduces the number of Key/Value heads at the model architecture level, lowering cache overhead. Amit's repository also lists KV Cache Compression separately, covering quantization, token eviction, cross-head sharing, and low-rank compression.

A commenter ran paired experiments with q4_0 and q8_0 on Qwen3.5-4B long context; in 9 out of 10 runs the answers were identical, and they asked whether the course covers KV cache quantization. This result can only be treated as a case study and cannot be directly generalized. Amit did not expand on this question in the reply, but the repository already has a KV Cache Compression article.

Throughput and latency are another line

Continuous Batching keeps the GPU as full of requests as possible at every decode step, instead of waiting for an entire static batch to finish. Speculative Decoding uses a small model to guess a few tokens first, then the large model verifies them; if the guesses are correct, time is saved. Token Streaming pushes generated results to the frontend as they are computed, so the user-perceived wait time is much shorter. Prompt Caching reuses the KV Cache of the same prefix, which is suitable for scenarios with fixed system prompts, multi-turn conversations, or agents.

These techniques do not solve the same problem. First-token latency, per-token output speed, overall throughput, and long-session VRAM cost each have their own corresponding optimization points. Stacking all the methods together is not necessarily faster.

Engines and hardware come later

The second half of the checklist moves into vLLM, SGLang, TensorRT-LLM, GGUF, Ollama, as well as GPUs, CUDA Kernels, TPUs, and LPUs. The core of vLLM is PagedAttention and continuous batching; the core of SGLang is RadixAttention's reuse of prefixes. A commenter asked whether vLLM or SGLang makes throughput easier to understand. Amit's checklist lists the two side by side, and the repository breaks down PagedAttention and RadixAttention separately.

For people with a development background, this checklist works well as a review index. For non-technical readers, it can also explain the technical layers behind product experience: first-token latency depends on Prefill; subsequent token output speed depends on Decode; long-session cost depends on KV Cache and PagedAttention; concurrency capacity depends on Continuous Batching and engine scheduling.

The resource repository Amit provides is here: https://github.com/amitshekhariitbhu/llm-inference-engineering . Original link: https://x.com/amitiitbhu/status/2115878569993969850 . The repository description says the series will continue to be updated.

If you remember only one conclusion: first distinguish Prefill and Decode, then look at which segment each optimization targets. Changing the GPU usually comes later.