500 tokens do not necessarily require 500 decoding steps
Avi Chawla clearly explains autoregressive and masked diffusion together: one continues token by token, the other fills in blanks in parallel. Worth noting is the engineering trade-off between KV cache and latency.
In the original tweet, Avi Chawla uses an LLM interview question to make a comparison: does generating 500 tokens necessarily require running 500 sequential decoding steps? The answer depends on how the model generates.
Regular users can think of a token as a small chunk of text that the model outputs. Both autoregressive LLMs and diffusion LLMs can use the transformer's embedding, attention, and feed-forward layers; the differences are in the attention pattern, training objective, and decoding process.
Autoregressive LLMs: one token at a time
In standard autoregressive decoding, each forward pass produces a probability distribution over the next token. The selected token is appended to the sequence, and generation continues. Causal attention prevents a position from seeing future positions, so the representations of previous tokens do not change when new tokens arrive. K and V can be cached and reused.
The cost lies in the number of steps. 500 tokens usually require 500 sequential token-generation steps. Techniques such as speculative decoding can reduce expensive model forward passes, but the basic rhythm of autoregression is still token-by-token.
Diffusion LLMs: fill in blanks in parallel starting from masked positions
In the masked diffusion approach Avi Chawla discusses, generation starts from the prompt and several masked output positions. A single denoising pass uses bidirectional context to predict candidate tokens for multiple positions. Confidence-based sampling keeps some predictions, leaves other positions masked, and fills them in the next round. The model gradually completes the whole sequence; it does not necessarily write a full draft first and then rewrite it token by token.
Some implementations generate in blocks rather than processing the entire response at once. Each step can finalize multiple tokens, so the number of steps can be fewer than the number of output tokens. The step budget affects speed and generation quality.
Fewer steps do not mean low latency
Bidirectional attention makes standard autoregressive KV reuse harder. The representation of a token may change as surrounding positions update, even if the token itself does not move. Whether caching can be done depends on the architecture and decoding strategy. Block diffusion can do KV caching across completed blocks.
Another easily overlooked point: processing more positions per step can increase total computation. Avi Chawla's reminder is direct: fewer steps do not guarantee low latency; total computation and engineering implementation both need to be accounted for.
Avi Chawla broke down why KV cache is critical in autoregressive models in another article: the first token is slow because the prefill stage must process the entire prompt and cache the K and V of each token in each layer. Once the cache is warm, subsequent tokens require only a single-token forward pass. The cost is GPU memory. The article gives the example of Qwen 2.5 72B: 80 layers, 32K context, hidden dim 8192, and a single-request KV cache can consume several GB of GPU memory. As concurrency rises, it often becomes a bottleneck earlier than the model weights. GQA, MQA, and Paged attention all address this cost.
In a reply, Saahith J included a 2D/3D visualization of GPT-2; you can turn on KV cache mode to see how the cache participates in generation: llm-visualized dot com.
Why causal attention cannot see the future: this short video explains it using causal masking. Without a mask, a sequence model can read the answer during training, making the training loss look great; after deployment, when the next event has not happened yet, the model has nothing to copy.
The questions in the comments are closer to engineering reality
Commenter Jerry Liu pointed out that when multiple tokens are unmasked per step, those tokens are sampled independently, so two tokens that should be consistent may conflict. The more tokens are committed at once, the more easily quality drops. Bidirectional attention also comes at the expense of KV cache.
Kushagra Mathur asked whether, for long code outputs, the number of steps for diffusion LLMs would rise again. Dharmendra Valiya was concerned with where the latency crossover point is for long outputs. Ricardo Dias gave a more practical criterion: fix the quality target and look at end-to-end latency. There may be fewer sequential rounds, but each round is wider, or reaching the same quality may require more denoising passes; the result may still be slower.
Autoregressive LLMs and diffusion LLMs are more like a division of labor, each constrained by caching, quality, and latency. Whether 500 tokens require 500 steps cannot be judged only by the generation paradigm; you also need to look at how much computation each step actually performs, how much cache can be reused, and where final quality lands.
发布时间: 2026-11-18 02:36