Local LLM 4x Faster—Not Through Hardware, but Through Speculative Decoding

On the same M5 MacBook Pro, with the same Qwen3.5 9B 4-bit model, llama.cpp reached 22.0 tokens/s, MLX reached 25.1 tokens/s, and an engine called uzu reached 92.1 tokens/s. Avi Chawla posted a video on X of the three engines running side by side, and the difference is visible at a glance.

Karpathy explained the physical limit behind this gap back in 2023. Every time an LLM generates a token, it has to read all the weights from memory and perform a full forward pass. The M5's memory bandwidth is about 153 GB/s, and a 4-bit 9B model takes up about 5.2 GB. 153 ÷ 5.2 ≈ 29. In other words, it can produce at most about 29 tokens per second, regardless of GPU compute. Both llama.cpp and MLX sit right at this upper bound. They are not doing anything wrong; ordinary decoding just produces only one token per pass.

uzu takes a different approach: make each forward pass produce several tokens. It uses a small model, DFlash, to draft the next 16 tokens, then uses a 56.7M-parameter adapter called Weaver to organize the draft into coherent candidate branches, and finally has the 9B main model verify the entire tree at once and accept the longest reasonable path among them. The main model is still the final decision-maker, and every output token is confirmed by it. In Avi's tests, the 9B model averaged 7.5 tokens per forward pass, versus 1 for ordinary decoding. That is why it can exceed the bandwidth limit: the weight-read cost amortized per token is only about one-seventh of the original.

uzu is an open-source inference engine from Mirai Labs, written in Rust, with Metal kernels, and supports Swift, Python, and TypeScript bindings. It also ships with a 4-bit checkpoint and uses a random Hadamard transform to handle outliers in quantization; on the M5 it runs on the GPU's int8 matrix path. Akshay wrote an in-depth breakdown (https://x.com/i/article/2108246386538094592) that explains these technical details in full.

If you want to verify it yourself, one command is enough:

brew install mirai
mirai --model trymirai/Qwen3.5-9B-M -m "Write a python function to merge two sorted lists"

The CLI stats Avi got: time to first token 0.12s, generation speed 98.10 tok/s, tokens per forward pass 8.39, memory usage 5.77GB. That means each forward pass produced an average of 8.39 tokens, so generation speed is close to 100 tok/s. uzu also provides an OpenAI-compatible local server, so existing apps can switch by changing the endpoint.

But 4x is not a free lunch. The gains from speculative decoding depend on how predictable the text is. For highly deterministic content such as code and math, the draft acceptance rate is high; open-ended chat is often rejected, and the speed advantage shrinks. Some users reproduced it on an M2 MacBook Air and found that uzu, MLX, and llama.cpp were almost the same speed—uzu's speculative configuration currently targets only the M5 series, and on older chips it falls back to ordinary decoding. In addition, the quantization formats of the three engines are not exactly the same, so this is a comparison of complete engine paths, not just a kernel comparison. Avi also acknowledges that 7.5 tokens/pass is an average; actual values fluctuate between code and open-ended text.

The bottleneck for local inference is memory bandwidth—this is already common knowledge. uzu's value lies in turning an engineering workaround for that bottleneck into a tool that ordinary developers can use directly. Hardware upgrades are expensive; software upgrades are free. This may be the most cost-effective way to speed up local LLMs right now.