The 10M-token attention layer spends 98% of its time finding tokens

On November 2, James Whittington announced on X one of the team's first open research results, Latent Sparse Attention (LSA). According to him, this is a step toward subquadratic attention, with the goal of matching full attention accuracy under ultra-long contexts.

The setup for sparse attention is not complicated: each query looks at only a small number of tokens, so the compute cost comes down. The trouble is that after 'looking at only a few,' you first need to know which few to look at. The longer the context, the more expensive this lookup becomes. The number Whittington gave is that at the 10M-token scale, searching for the correct tokens accounts for 98% of the attention layer's time. Matrix multiplication has long ceased to be the bottleneck; indexing is.

LSA changes the lookup method. From the published video, ordinary sparse attention evaluates each key-value pair one by one; LSA first groups keys into clusters, scores the clusters, and then scores the keys within the selected clusters. The lookup goes from one level to two, and the number of objects that need scoring drops by a large margin.

Performance numbers: LSA makes sparse attention up to 22.7x faster and full attention up to 72.4x faster. The bottom of the video also compares index memory: 0.32 GiB for sparse attention and 0.24 GiB for LSA. The test is based on one attention layer of GLM-5.3-Flash, with a 10M-token prefill, and the caption notes that only H2O0 was used.

There are several boundaries to keep in mind with this result. It is a measurement on a single attention layer, not a complete end-to-end training or inference number. Both 22.7x and 72.4x are marked 'up to,' and the specifics vary with context length and configuration. Whether accuracy can reliably match full attention also needs verification on more tasks. What is public now is a research release, not yet a product promise.

Still, the direction it points to is worth noting. Over the past two years, discussions of long context have often centered on KV cache compression, attention kernel functions, and engineering optimizations such as FlashAttention, with most work focused on FLOPs and memory bandwidth. LSA shifts the lens to index search: when context reaches the tens of millions, the main cost in each attention layer becomes 'where to look,' while the computation itself recedes into the background. This observation is more informative than any particular speedup ratio.

Also involved in this work are @kubicek_ales, @caciolai, @PengLiangzu, @SukritSumant, @Ben_Hoov, @FraserGreenlee, @RMBattleday, @radim_urban, @thoefler, and Whittington himself. Whether the code and paper are released at the same time is not covered in the tweet and will have to wait for follow-up.

On the road to long context, everyone can do the compute math, but not many have seriously done the indexing math. LSA at least puts the numbers for the latter on the table.