Qwen3.8-27B + a pi-specific LoRA: SWE-bench Pro rises from 39.2% to 64.7%, using 4.5× fewer tokens
Hugging Face trained a Qwen3.8-27B-specific LoRA for the pi coding agent. On SWE-bench Pro hard 51, it goes from 39.2% to 64.7%, with half as many tool calls. With thinking turned off, it matches the base model with xhigh enabled while using 4.5× fewer tokens. It has only been validated on pi.
Hugging Face's harness-hub has released a LoRA adapter specifically for Qwen3.8-27B used with the pi coding agent. The numbers: SWE-bench Pro hard 51 goes from 39.2% to 64.7%, Terminal-Bench 2.1 goes from 39.5% to 55.8%, and tool calls per task drop by 55% to 62%.
One number needs clarifying first. Clement Delangue wrote 29.4% to 62.7% in the tweet, but the model card specifically explains that in the earlier set of data, 9 rollouts of the base model were marked as failures because the tunnel between the sandbox and the model server dropped midway. After correcting and rerunning, the numbers on both sides are 39.2% and 64.7%. It's not a small change, but the conclusion is unchanged.
This adapter solves a very specific problem. The same base model performs differently in different agent harnesses, and each harness has its own tool names, parameter formats, and interaction flow. When you run a general-purpose model in pi, the model has to learn this setup on the fly, and the cost shows up in the number of tool calls and in accuracy. Hugging Face's approach is to use reinforcement learning to train pi's interaction patterns into a LoRA.
The method is called AsyncGRPO, with LoRA rank 32 and alpha 64, attached to the linear layers. Training used 750 scale800 tasks plus 300 TermGrade tasks. The reward is the task verifier's correctness plus a tool-efficiency reward weighted at 0.1. It ran for 160 steps on 2 B200s.

The most interesting finding is the thinking mode. With thinking turned off, the adapter matches the base model with xhigh enabled in accuracy: SWE-bench Pro 64.7% vs. 64.7%, and Terminal-Bench 56.6% vs. 54.2%. But it generates about 4.5× fewer tokens, and the number of model calls drops from 48 to 21.
What training consumes is the thinking process, compressing reasoning ability into direct tool operations. The base model needs to think at length before acting; the adapter acts directly.
The tool-call efficiency gains are also substantial. On tasks that both can solve, the adapter averages 23 turns versus 59 for the base model. Compared with the base model with thinking enabled, it's even more pronounced: 5.1k generated tokens vs. 34.7k.

A concrete example: in SWE-bench Pro, when fixing the human_to_bytes function, the base model repeatedly searches and reruns code snippets, hits the 100-step limit, and scores 0. The adapter finds the function, changes it, runs the tests, fixes the last failing case, and finishes in 8 tool calls, scoring 1.

The limitations are also clear. It must be attached to Qwen3.8-27B and cannot be swapped to another base model. Training and evaluation were both done in pi, and the official team has not provided data on how it performs in Claude Code or Codex. Under the tweet, both Charlie and Asmir asked this question; the official account only replied to Asmir, asking whether he wanted to try training one for them.
eishan asked more directly: how can you be sure the model hasn't memorized the answers to these public benchmarks? The official reply was that yb2698 confirmed it. The model card also addresses this: there are no hidden tests in the training data for the base model and adapter, and evaluation uses the hard 51 task set. But the benchmark itself is public, so whether the pretraining data has seen it can only be self-checked by the training party.
There are two more pieces of context worth adding.
This kind of thing is called a harness adapter. The idea is one base model paired with multiple adapters, each corresponding to an agent harness. The benefit is low cost: a single vLLM instance can serve multiple adapters at once, and neither training nor evaluation requires merging weights.
The blog mentions a safety mechanism called safeagents. It is a reverse proxy that runs every tool call through a rule check before execution. The rules are hard-coded regular expressions, not a model, and the decision time is 10 to 20 milliseconds. It guards against reward hacking, such as the model reading the tests directory, rewriting the grader, or downloading the original dataset from the Hub to find answers. TwIL-8B tried to use huggingface_hub at step 0 to find the task's source data, was blocked, and that rollout scored 0.
The training-data distribution is also interesting. Of 270 tasks, 253 are build-type tasks: building a project from scratch. There are only 11 repair tasks, 6 extend tasks, and no qa tasks at all. This relates to the composition of open-source agent trajectories: most shared sessions are one-off project-building requests. The blog says directly that to train a better adapter, what's most lacking is sessions where something breaks and then gets fixed, as well as sessions where a project gradually grows across multiple requests.

The model card page says Apache-2.0, and the weights can be downloaded directly. The training code is in TRL's examples/async_grpo_harbor directory, and the blog provides the complete environment-generation method. To train an adapter for your own harness, the steps are roughly: save the session trajectories from when you normally use the agent, use safeagents scrub to clean sensitive information, use Repo2RLEnv to convert them into verifiable tasks, and run the same RL pipeline.
Hugging Face says this pipeline is not specific to their harness. That sounds like a pleasantry, but if you use the same agent for work every day, the idea really does hold up.
发布时间: 2026-11-18 23:39