Coding Agent Harnesses: Study Finds They Buy Tokens, Not Wins

Changing a coding agent’s harness flipped the outcome on 13% of hard SWE-bench tasks, exactly as many as rerunning the same setup with nothing changed. That is the core result of a new study from two Shandong University researchers, and it cuts against a steady stream of vendor claims about harness-driven score gains. For engineering teams choosing a coding agent, the paper says the harness mostly decides the bill, not the pass rate.

What Happened

The Decoder reports on the paper, titled “What Does a Harness Buy? Tokens, Mostly”, by Yangze Liu and Zhongyi Han. They ran five models, Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash and HY4-Preview, through three shipped harnesses: Claude Code, OpenCode and mini-SWE-agent, which gives the model only a Bash tool and no context management. Claude Opus 5 ran in Claude Code as a reference. Tests used the 45 hardest SWE-bench Verified tasks plus a pool of 447 more, and several setups were rerun to measure noise.

All runs were offline. In early tests with web access, 71% of Deepseek and Qwen attempts pulled the correct fix straight from GitHub, so the researchers cut off web access. They don’t claim any maker trained on the test data.

Why It Matters

The study’s key move is measuring noise before comparing harnesses. Reruns of the same setup already vary by up to six points on SWE-bench Verified, according to the authors. On the 447-task pool, Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, were equivalent within five points. With 45 tasks, even a 13-point gap shows up only half the time, so the 2.4-point lead Zhipu AI advertises for its own harness over Claude Code falls below what either sample can detect.

The one harness effect that cleared the noise was a loss. OpenCode trailed by up to nine points on the large pool, and about half of that gap traced to an output limit: when a model hit it mid-step, Claude Code prompted it to continue, while OpenCode let the step end.

Cost is where harnesses really differ. With the same model, tasks and price list, cost per task varied by up to 3x. The gap comes from what each harness resends on every step. System prompt and tool descriptions run about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode and under 1,000 for mini-SWE-agent. That preamble sets the baseline and the step count multiplies it. With GLM-5.3-Flash, Claude Code cost nearly twice as much as mini-SWE-agent. With Deepseek-V4-Flash, mini-SWE-agent needed so many extra steps that the two came out even.

What This Means for Builders

Thinking was the only setting where harness choice changed outcomes: without it, mini-SWE-agent with Qwen3.6 dropped eleven points while the other two harnesses stayed within noise. The authors say their grid is too small to explain why, and the study relies mainly on one benchmark, with a small Terminal-Bench 2.1 cross-check showing the same pattern. Opus 5 ran only in Claude Code, so the study can’t show how a closed frontier model behaves across harnesses.

The finding fits an earlier comparison Stack Archive covered, in which framework choice nearly tripled costs for the same model. The authors recommend publishing a rerun of the same setup alongside every harness comparison, and they’ve released their data and scripts.

Source: The Decoder

Content created with AI assistance and reviewed for accuracy.