Wink Pings

The Same Qwen3.8-27B with a LoRA Adapter Jumps from 29% to 62% on SWE-bench—and Uses 55% Fewer Tool Calls

harness-hub releases Qwen3.8-27B-pi-adapted, a LoRA adapter trained with RL specifically for the pi coding agent. SWE-bench Pro rises from 29.4% to 62.7%, Terminal-Bench 2.1 from 39.5% to 55.8%, and tool calls drop by 55% to 62%. Open-source weights, Apache 2.0. What's more interesting is the pipeline behind it: one base model with multiple adapters, switched by agent framework—and you can reproduce it yourself.

There is a team called harness-hub on Hugging Face that released a LoRA adapter, Qwen3.8-27B-pi-adapted. What it does is very specific: it fine-tunes the general-purpose Qwen3.8-27B model with reinforcement learning for the pi coding agent.

pi is a coding agent from Earendil Works, with its own toolset, prompts, and control flow. The same model performs differently when placed in different agent frameworks because the tool-call formats and context-management approaches differ. harness-hub's approach is to train a small separate adapter for each agent while keeping the base model unchanged.

banner

First, the numbers. Evaluated under pi, pass@1:

  • SWE-bench Pro (hard 51): 29.4% to 62.7%, a 33.3-percentage-point improvement
  • Terminal-Bench 2.1: 39.5% to 55.8%, a 16.3-percentage-point improvement

Tool calls also dropped. On SWE-bench, there are on average 55% fewer calls per task, and on Terminal-Bench, 62% fewer. On tasks solved in both settings, the number of tool calls was 59% and 48% lower, respectively.

结果对比

The official release provides a comparison trajectory. For the same SWE-bench Pro task—fixing Ansible's human_to_bytes—the base model searches around, repeatedly reruns snippets, and still fails after hitting the 100-step limit. The adapted model finds the function, changes it, runs the tests, fixes the last case, and finishes in 8 tool calls.

轨迹对比

Training used AsyncGRPO, TRL's experimental implementation, with LoRA rank 32 applied to the q, k, v, o, gate, up, and down projections. Learning rate was 2e-5, with 8 rollouts per task, temperature 0.8, and thinking disabled. The hardware was 2 NVIDIA B200s—one running vLLM inference and one running the trainer.

The data deserves a separate look. Tasks came from the Harbor environment, rollouts ran in HF sandboxes, and the training set mixed 750 scale800 tasks with 300 TermGrade tasks. Every tool call was checked by safeagents. safeagents is a reverse proxy sitting between the agent and vLLM, using rules to determine whether a call is compliant: it cannot edit test files, cannot write reward files, cannot access the public dataset containing the task answer, and cannot curl a host that never appeared in the conversation.

The blog mentions one detail. At step 0, before any updates, TwIL-8B was given a data Q&A task. Its first reaction was not to compute the answer, but to use huggingface_hub to dig through the bucket where the original notebook was stored. safeagents blocked this call, and the rollout was scored 0. RL training can lead models to find shortcuts, so the safety rules are enabled from the very first step.

训练曲线

Using it isn't complicated. Load the adapter with vLLM, and in the pi configuration set the api to openai-completions; if you don't, pi will use the Responses API, and the adapter was not trained on that interface. The official release also provides GGUF versions for transformers serve, llama.cpp, and Llama.app.

The adapter itself is of course useful. But what's more worth looking at is the pipeline behind it: one base model, multiple adapters, each trained for a different agent framework, sharing the same base weights, all served by a single vLLM instance. Training cost is much lower than full fine-tuning, and at deployment you select the adapter by agent.

harness-hub has open-sourced this pipeline. Trajectory data comes from public agent sessions on the Hub, converted into verifiable tasks with Repo2RLEnv, checked for safety with safeagents, and used to run AsyncGRPO with TRL. The sessions your agent runs every day can go through the same path to train your own adapter.

发布时间: 2026-11-18 10:29