32GB of RAM runs a 125B model: Unsloth's quantization loses only 20% accuracy

Unsloth has released a new GGUF for Qwen3.8-Flash-Next, re-quantized with the Dynamic V3.1 algorithm. Official data: it is 15% more accurate than other frontier quantization schemes, with the highest scores on agentic benchmarks such as Terminal Bench 2.1, DeepSWE v1.1, and Automation Bench.

Running the benchmarks burned 26.2B tokens: DeepSWE accounted for 18.1B and Terminal Bench for 5.2B. Sampling parameters were fixed: temperature 1.0, top_p 0.95, top_k 20, min_p 0, with no presence or repetition penalty. Context was set to 256K (non-YaRN); general non-agentic benchmarks used 128K.

1-bit 75GB, 80% top-1 accuracy

The smallest 1-bit quantized version is 75GB, 79% smaller than BF16's 355GB, and retains 80% top-1 accuracy. Hardware requirements are as follows:

In the new architecture, the N-gram embedding layer (PLE) is only quantized down to 4-bit at minimum, because it uses a random-access pattern; cutting it too aggressively would damage the model. As a result, the 1-bit version's size isn't squeezed to the absolute limit, but the accuracy loss is actually narrowed.

A commenter made a comparison: IQ1_M scored 69.5 on Terminal Bench 2.1, and IQ3_XXS scored 70.8, both above the BF16 baseline of 68.3. He also asked how many runs were done per data point. When a quantized version beats the original model on a single benchmark, it usually means the test variance is larger than the gap. This observation is worth noting, but it doesn't change the overall trend: on most benchmarks, the quantized versions are still below BF16.

MTP acceleration: 1.3x to 1.7x

MTP (Multi-Token Prediction) raises inference speed from 100 tokens/s to 170 tokens/s on an RTX 6000 PRO. Older Macs have lower memory bandwidth, so the gains will be smaller. Unsloth Desktop enables MTP by default; for llama.cpp, you need to download the MTP files separately and compile with a custom branch.

Leave an extra 1–2GB of memory for MTP. 3-bit quantization plus MTP needs 91GB of memory.

The engram layer's precision can be switched

Q4_K_XL comes with an IQ4_NL engram table (28.8GB). If you have enough disk space, you can switch to Q8_0 (54.4GB, relative RMSE 0.0053) or even BF16 (102.4GB, RMSE 0). Use the set_split.py script in the repo to modify the shard header info; after renaming, replace the original table shards.

Running it

Option 1: Unsloth Desktop, supporting macOS, Windows, and Linux. Search for Qwen3.8-Flash, choose the quantization version you want, download, and run. Parameters are configured automatically, and MTP is enabled by default.

Option 2: command line:

unsloth run --model unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL

Or use llama.cpp; the model card provides full download and launch commands. Configurations for Ollama, vLLM, LM Studio, and Docker are also available.

The model itself is a 125B-parameter MoE architecture with 6B active parameters, plus 51B n-gram embeddings and 4B MTP, with native 262K context support. The 1-bit version runs on 32GB-RAM devices by using mmap to put the engram layer on SSD. Running a 125B-parameter multimodal model on a 32GB-RAM device would have been unrealistic two years ago. The cost is 20% accuracy.

Original tweet Model download Run documentation