Ollama Decoding Is 2.7× Slower: Magnitude Moves Kernel Tuning to the Local Machine

Akshay posted a set of local inference comparisons: base M5 MacBook Pro, Qwen3.5 4B 4-bit, the same 9.2k-token prompt. Ollama was running an MTP build, and both sides used speculative decoding. As a result, decode went from 22.6 tok/s to 60.0 tok/s, a 2.7× difference; prefill went from 464 tok/s to 755 tok/s, a 1.6× difference; end-to-end time was cut in half.

Akshay recorded the two runs side by side:

Ollama’s numbers don’t mean it performs poorly. Its kernels are tuned ahead of time. A kernel is the small program behind each matrix multiplication and attention step on the GPU, and its speed depends on how work is divided among GPU cores and how data moves through memory. Apple Silicon ranges from 8 to 80 GPU cores, bandwidth from under 70 to over 800 GB/s, and each model has different layer sizes. A prebuilt engine has to choose one set of settings for all chips, and most local environments run within that compromise.

Magnitude takes a different approach: when a model is downloaded, it compiles that model’s kernels on the local machine. It tries different kernel configurations and keeps the fastest set on this chip; then it pairs each model with the fastest speculative-decoding drafter it can find, so each model forward pass produces several additional tokens.

This matters more for agents. Every file and tool output has to be processed before the model starts writing, so kernel efficiency in the prefill stage shows up directly. Concurrent sessions can also share the prefix cache, preventing parallel agents from slowing each other down.

But the questions in the comments are worth laying out, too. Some commenters noted that 16GB of memory is tight for a model plus a long prompt; others asked whether a 4B 4-bit model would swap on an 8GB MacBook Air. Someone wanted to see the same test with Qwen 3.8 27B Q8 on an M5 Max with 128GB. Others asked how much of the 2.7× comes from the drafter, what the draft acceptance rate is, whether shared memory bandwidth would become a bottleneck with four concurrent agents, how long first-time tuning takes, and whether peak memory changes. Someone compared Magnitude’s approach to Gentoo: kernels compiled for your chip, and the old-school approach wins. These follow-up questions don’t invalidate the results, but they separate single-run, single-stream numbers from everyday agent use.

Magnitude is open source under Apache 2.0, written in Rust, and available as both a desktop app and a CLI. It can connect to agents such as Claude Code, Codex, OpenCode, and Pi with one click. It supports Apple Silicon, NVIDIA, AMD, and Intel, and can even run on CPU alone. The repository is on GitHub: http://github.com/magnitudedev/magnitude

Install command:

curl -fsSL https://magnitude.dev/install.sh | sh

Windows PowerShell:

irm https://magnitude.dev/install.ps1 | iex

The hidden tax of local inference is often described as model size, but it also comes from kernel-to-chip matching. Precompiled one-size-fits-all builds have a cost, and automatic local tuning moves that cost to download time. Don’t treat single-run, single-stream numbers as the answer for the full agent workflow; wait for tests with larger models, concurrency, and long prompts.