Running 27B Models at 2-bit Quantization for Tool Calling: These Real-World Benchmark Results Are Worth a Look

Fitting a 27B-parameter model on a consumer-grade GPU is already a tight squeeze. When you mention pushing it down to 2-bit quantization, most people's first reaction is: can this even function properly? The combination of Underdog's Saluki 27B and ISTA's GSQ-RCO quantization gives a clear answer: it is totally worth a try, as long as you use it for structured tasks such as tool calling.

First, let's clarify what each of these components is. Saluki 27B is a model released by Underdog, with the base built on Qwen3.8-27B. GSQ-RCO is a dedicated quantization method from ISTA, with strong targeted optimization that follows a task-specific tuning path, which is a completely different line of thinking from generic one-size-fits-all quantization solutions. The llama.cpp used here is the official stock version, so you don't need to wait for custom branches or apply patches on your own.

The metadata in the GGUF header is well worth exploring. Quantization version, tensor type, and per-layer configuration are all recorded there. Checking the header first after you get the file can save you a lot of trial and error time. Memory usage is genuinely reduced: the weight size shrinks drastically under 2-bit quantization, and you need to turn on the KV cache quantization option at the same time, otherwise your VRAM will still overflow once the context gets long.

The BFCL tool calling score is the core focus of this guide. Tool calling has high requirements for format stability. After quantization to 2-bit, whether the model can correctly preserve the function name and parameter structure directly determines whether this solution is usable. In real-world tests, Saluki's performance at this quantization level falls in the usable range: complex multi-turn calls have minor performance fluctuations, while simple single-turn calls are mostly stable.

Running it on stock llama.cpp is not complicated: pull the code, compile, load the GGUF file, start the server, follow the regular workflow. It is worth noting that do not max out the context length at the very beginning. The quality degradation for long context under 2-bit is far more obvious than that under 4-bit, so set the length according to the actual needs of your task first. Make sure the tool calling template and chat template are aligned: for Qwen-series models, one wrong marker in the format will mess up the entire output.

This combination has a clear positioning: it is designed for scenarios with limited VRAM that require local tool calling deployment. If you want top-tier general conversation quality, 2-bit quantization is not the right choice. But if you want to fit a 27B model into your workflow without upgrading your hardware, Saluki plus GSQ-RCO is well worth half an hour of testing.