Wink Pings

Generating 500 tokens does not necessarily require 500 steps

Avi Chawla discusses autoregressive LLMs and masked diffusion LLMs together. The difference is not only in decoding order, but also in KV cache, attention, and latency accounting.

Avi Chawla explained on X a question that often comes up in interviews: what exactly is the difference between autoregressive LLMs and diffusion LLMs? First, remember one sentence: generating 500 tokens does not necessarily require 500 sequential decoding steps. A token can be understood as the basic unit of text that a model outputs.

The LLMs most people are familiar with are autoregressive. Each forward pass gives the probability distribution for the next token; one is sampled, appended to the input, and the process continues. Causal attention prevents a position from seeing future positions, so the representation of earlier tokens does not change just because new tokens arrive later. This property is crucial: it lets the K and V of each layer be cached and reused. Generating 500 tokens is still usually 500 sequential generation steps. Speculative decoding can reduce the number of forward passes through the expensive model, but token-by-token sequential generation remains the default path.

Masked diffusion starts from a different point. Output positions are first masked and enter the model together with the prompt. One denoising pass uses bidirectional context to predict candidate tokens for multiple positions. Confidence sampling keeps some predictions, while other positions remain masked and are filled in the next round. The model gradually completes the sequence; it does not necessarily write a full draft first and then rewrite it word by word. Some implementations generate in blocks and do not have to process the entire answer at once. Multiple tokens can be finalized in one step, so the number of steps may be less than the number of output tokens. The step budget affects the balance between speed and quality.

Both can use transformer embeddings, attention, and feed-forward layers; the difference lies in the attention pattern and training objective.

Demo video attached to Avi Chawla's original post:

There is an engineering accounting detail here that is easy to overlook. Bidirectional attention makes standard autoregressive KV reuse harder. A token's representation changes when surrounding positions are updated, even if the token itself has not moved. Whether caching can be done depends on the architecture and decoding strategy. Block diffusion can perform KV caching across completed blocks. A lower step count also does not automatically equal low latency, because one step may process many positions; total computation and implementation quality both need to be factored in. Bidirectional conditioning also has a direct use: filling gaps in text or code.

In another long article on KV caching, Avi Chawla breaks down the sources of latency very clearly. After a user sends a prompt, the model first performs one forward pass to compute and cache the K and V for the entire input; this is prefill, and it is the most compute-intensive part. Once the cache is warm, each new token only needs a forward pass for one token. The wait for the first token is called TTFT; the longer the prompt, the longer the prefill. In a reply, Andika Rachman compared this to: prefill is like reading the entire meeting transcript, while decoding is like starting to participate in the conversation. TTFT is slow because the model is spending time catching up on context.

The cost is GPU memory. K and V must be stored for every token at every layer. Avi Chawla gave the example of Qwen 2.5 72B: 80 layers, 32K context, hidden dim 8192; the KV cache for a single request can consume several GB of GPU memory. With hundreds of concurrent requests, the cache often takes up more space than the model weights. GQA and MQA reduce memory by sharing key/value heads, with as little quality loss as possible. If context length doubles, the KV cache for a single request also doubles, and how many users can be served concurrently must be recalculated. Work such as paged attention is aimed squarely at GPU memory utilization.

In a reply, Saahith J posted a 3D and 2D visualization of GPT-2; turning on KV-cache mode in the settings lets you compare and see the effect of caching. The two images from the original post are below.

GPT-2 可视化 1

GPT-2 可视化 2

A more worthwhile judgment is this: it is not a simple matter of autoregression replacing diffusion or vice versa. Autoregressive KV caching and serving stacks have been refined for many years, and the engineering path is mature. The diffusion route is attractive for parallel decoding, gap filling, and fewer steps, but cache reuse, per-step computation, and quality stability all still need to be validated in real serving. Counting only decoding steps easily leads to pretty conclusions; what actually determines the experience is the three ledgers of latency, throughput, and GPU memory.

Original post: https://x.com/DailyDoseOfDS_/status/2122641405134991645

KV caching long article: http://x.com/i/article/2034896077460316163

发布时间: 2026-11-17 21:28