13 Attention Mechanisms: Better to Sort by Bottleneck Than Memorize by Name

Attention mechanism names often get piled into one list. MHA, MQA, GQA, MLA, FlashAttention, and PagedAttention lined up together look like things on the same layer, but each gets stuck at a different place. Daily Dose of Data Science lists 13 of them and divides them into four categories by bottleneck. This way of classifying is more useful than memorizing them by paper date.

Someone in the comments added: KV cache and masking are the easiest to confuse. This classification happens to separate the two.

1. KV cache size is the bottleneck

During generation, the model caches key-value states to avoid recomputing at every step. The larger the cache, the tighter the GPU memory.

• MHA: Each query head has its own key-value head. Flexible, with the largest cache.
• MQA: All query heads share one key-value head. The cache becomes smaller.
• GQA: Query heads are grouped, and each group shares one key-value head. Balances memory and model quality.
• MLA: Before caching, keys and values are compressed into a smaller latent representation, further saving KV memory.

2. Which tokens the model can see is the bottleneck

• Causal attention: Each token can only look backward. Used for autoregressive generation.
• Bidirectional attention: Can look in both directions, suitable for understanding the complete input at once.
• Sliding Window Attention: Each token only looks at nearby context, not across the entire sequence.
• StreamingLLM: Keeps a small number of anchor tokens plus recent context, allowing generation to continue within limited memory.

3. The attention computation itself is expensive

• FlashAttention: Splits exact attention into small blocks and computes them in fast on-chip memory, reducing expensive GPU memory transfers.
• Sparse Attention: Skips some token-to-token connections, computing less attention.

4. Online serving throughput is the bottleneck

• PagedAttention: Stores the KV cache in blocks on demand, reducing GPU memory waste and fragmentation.
• RadixAttention: Organizes shared prefixes so multiple requests can reuse KV states without recomputation.
• Prefix Caching: KV blocks for repeated prefixes such as system prompts are reused directly, reducing repeated prefill.

The original emphasizes that these techniques are complementary. An LLM can simultaneously use GQA to shrink the KV cache, FlashAttention to compute attention efficiently, Sliding Window Attention to limit context interaction, and PagedAttention to manage the cache in production. After classifying by bottleneck, the 13 names no longer feel like a pile of paper labels.

The video and full breakdown come from the Daily Dose of Data Science tweet. Akshay wrote a longer article: original tweet, Akshay's article, quoted tweet.