
Harness Adapters: How a LoRA Tunes an Open Model to Your Agent
Agentic AIQwen3.8-27B solves 39% of Terminal-Bench 2.1 tasks inside the pi coding agent. With its weights frozen and a small RL-trained LoRA added, it solves 55% while using about half the tokens. Hugging Face’s argument is that general-purpose models leave performance on the table in your specific harness. For teams running open models in coding agents, that turns harness choice into something you can train for, not just pick.
How It Works
Hugging Face’s harness adapters post trains one LoRA per harness, for pi, Hermes Agent and OpenCode, with reinforcement learning run inside the harness itself. Most RL frameworks train against a re-implementation of the agent loop. Here the harness runs exactly as it ships, and a proxy between it and vLLM records the exact tokens it produced for the trainer. Training uses TRL’s AsyncGRPOTrainer, which doesn’t wait for every rollout to finish before the next optimization step.
Task validation. RL needs tasks, a verifier and evidence the verifier is right. The team pulled Harbor-format tasks from SWE-smith and SWE-Gym and generated more from merged pull requests with Tasksmith. Then they validated each one: the reference solution must score highest, doing nothing must score clearly lower, and replays must be deterministic. 858 of 1,127 Hub candidates passed. The checks caught broken verifiers, including a Terminal-Bench 2.1 one that accepted known-wrong answers.
Difficulty filtering. GRPO scores each attempt against others on the same task, so a task the model always or never solves teaches nothing. Of 442 scored tasks, Qwen3.8-27B solved 151 every time and 120 never. Only the 171 it solved sometimes qualified. The final reward was verifier correctness plus a small 0.1 tool-efficiency bonus, and a safeagents plugin blocked 647 of 1.3 million tool calls during training.
Results. Each adapter beat the base model in its own harness. On the 51 hardest SWE-bench Pro tasks, pi rose from 39.2% to 64.7%, OpenCode from 35.3% to 51.0% and Hermes from 29.4% to 43.1%. On tasks both solved, the adapters made 53 to 72% fewer tool calls. Merging the three LoRAs at half weight produced a super-adapter that beat the base model under all three harnesses.
What This Means for You
The efficiency gain may matter more than accuracy for anyone paying for inference. Fewer tool calls and tokens per solved task mean cheaper, faster agent runs, a useful counterpoint to the study finding that coding agent harnesses buy tokens, not wins.
There are limits. Hugging Face warns that you can’t compare scores across harnesses, since each was run with its default setup, and the super-adapter’s half weight was chosen on Terminal-Bench, making those numbers slightly optimistic. The specialists also lead with thinking off. If you use one harness and keep thinking off, Hugging Face says to take that harness’s specialist. The adapters, tasks and plugin are published so others can reproduce the recipe.
The So What
Teams running an open-weight model inside one coding agent should try the published adapter for their harness before paying for a larger model, and measure tokens per solved task, not just pass rate. Teams with their own task distribution should budget first for validated, mid-difficulty tasks. That filtering is the part of the recipe Hugging Face spent the most effort on.
Source: Hugging Face
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →