Linear attention has to move state every token it generates; LeapQuant compresses it to 8-bit without losing accuracy

Model families such as Qwen, Kimi, and GLM are replacing part of standard attention with linear attention. Linear attention compresses context into a fixed-size recurrent state, eliminating the growth of attention computation and KV cache in long contexts. The cost is in decoding: for every token generated, the linear attention layer must read the recurrent state from HBM, perform a lightweight update, and write it back. State reads and writes become the bottleneck. With prefix caching enabled, the serving system also has to keep a copy of the state for each cached prefix, which increases GPU memory usage.

Yi Pan introduced LeapQuant in a tweet. The tweet claims it is the first training-free method that can run 8-bit or 4-bit recurrent states, covering Qwen, Kimi, and GLM. The paper's authors also include Haocheng Xi, Kan Zhu, Xingyang Li, Song Han, Kurt Keutzer, Ion Stoica, and others. The project page's conclusions are direct: accuracy is close to FP32, kernel speedups average 2.05x to 3.70x, and end-to-end inference averages 1.57x.

Quantization looks like a natural choice, but the difficulty is that the recurrent state updates at every step. Quantizing at every step introduces rounding error into all subsequent states, and the error accumulates in long contexts. The state also has outlier rows and columns that expand the quantization range, pushing ordinary values onto coarser quantization grids. Yi Pan notes that naive FP8/INT8 collapses, and BF16 also scores about 11.5 points lower than FP32 on average on AIME 2026, BeyondAIME, GPQA-Diamond, and LiveCodeBench v6.

LeapQuant uses three techniques to address these two problems.

The first is per-window quantization. The state is not re-quantized after every token; instead it crosses a token window and is quantized only once at the end of the window. Within the window, the low-precision state stays fixed, token updates are first cached at high precision, and the output is computed from the fixed state plus the cached updates. The tweet mentions cycling every 16 tokens. With fewer quantization steps, the error no longer rolls over at every step.

The second is Compensator Tokens. The largest outliers in the state are extracted and kept as several high-precision rank-one outer products. They have the same form as real token updates, so they can be merged directly into the window's update path with almost no additional state updates. The remaining residual is easier to quantize.

The third is residual smoothing. After removing the compensator tokens, the magnitude of the residual can still be uneven across different key channels. Before quantization, the magnitude of each row is balanced first; the smoothing scale is fixed within the window and refreshed at window boundaries. The project page gives an example from a Qwen3.5-9B attention head: outliers drop from 176x the median to 5.3x.

Results averaged across the three models: 8-bit and 6-bit LeapQuant both score 84.4, FP32 scores 84.0; BF16 scores 72.5, FP8 scores 45.1, and INT8 scores 47.7. 4-bit LeapQuant averages 80.6, 3.4 points lower than FP32, but far above the other 4-bit methods in the table.

On speed, the tweet gives a maximum kernel speedup of 4.25x, while the paper and project page give an average range of 2.05x to 3.70x. The project page and GitHub say an end-to-end average of 1.57x, while the arXiv abstract says 1.47x; there are differences between versions. On memory, the project page says recurrent-state memory traffic drops 3.4x and end-to-end GPU memory drops by up to 56%. The tweet also says inference throughput is up to 2x.

Code is already open source and integrated into vLLM 0.30.0 as a plugin, supporting Qwen's Gated DeltaNet and GLM's Kimi Delta Attention. Usage is straightforward:

git clone https://github.com/leapquant/LeapQuant.git
cd LeapQuant && pip install -e .

bash serve.sh <model>

serve.sh launches an OpenAI-compatible vLLM service with 8-bit recurrent state, passing extra arguments to vllm serve. The paper, code, and project page are here:

Paper / Code / Project page

The value of this kind of work is not in a single benchmark score, but in reducing state storage costs in long-context and prefix caching scenarios, letting the same GPU memory serve more prefixes. A commenter, Steven Gao, mentioned that it might help with prefix reuse; Yi Pan replied that this is one of the sources of the memory reduction and throughput gains. Real-world gains still depend on model architecture, batch size, context length, and kernel implementation; now that the code is open source, you can test all of these yourself.