A preprint posted to arXiv on October 3, 2026 asks a question that coding-agent vendors rarely answer: when the underlying language model stays the same, how much does the software wrapped around it change the score? The paper, by Yangze Liu and Zhongyi Han, argues that the answer is “much less than advertised,” and that the extra machinery mostly increases the number of tokens an agent spends. It has not been peer reviewed, so its numbers should be read as a careful preprint, not settled fact. This article walks through what the authors did, what they found and what the findings do and do not show.
What a harness is and why it matters
A coding agent is a language model plus a harness. The harness is everything that turns a chat model into something that can work inside a software repository: the system prompt, the set of tools (shell, file editor, search), and the context management that summarizes or trims history as a task grows. Products such as Claude Code and OpenCode are harnesses, and so is the deliberately minimal mini-SWE-agent, which gives the model only a Bash tool and no history compression.
Vendors often advertise that a new harness release lifts pass rates, and public leaderboards mix harnesses and models freely. If harness changes really moved scores by 20 points, that would be a major engineering lever. The paper’s contribution is to measure that lever with the model held constant, and to measure how much a score moves when nothing at all changes except the run.
How the study was run
The authors ran five open-weight or preview models through three production harnesses in their shipped versions: Claude Code, OpenCode and mini-SWE-agent. The models were Qwen3.6, Qwen3.8, DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview. Claude Opus 5 was included only as a reference inside Claude Code. Evaluation used SWE-bench Verified: the 45 hardest tasks, plus a pool of 447 more. Several configurations were run twice so the team could estimate pure run-to-run noise.
All final runs were offline. According to the study as summarized by The Decoder, early tests with web access showed that 71 percent of attempts by DeepSeek and Qwen models retrieved the correct fix from GitHub rather than deriving it. Claude models almost never did this. The authors discarded those runs and say they do not claim any developer trained on test data; they simply cut off web access for the rest of the experiments.
What they found
The headline result is that harness choice is comparable to noise. On the 447-task pool, with the two models run there, Claude Code (the heaviest harness) and mini-SWE-agent (the lightest) are equivalent within five points. Across the three harnesses, token consumption was two to three times higher than the leanest setup for each model, yet pass rates stayed within measurement uncertainty.
On the hard 45-task subset, harnesses differed by two to five solved tasks per model, while simply rerunning the same configuration moved the count by up to three. The task-level picture is similar: between two reruns, the median outcome flips on about 13 percent of tasks; between two different harnesses, it also flips on about 13 percent; between two different models, about 22 percent. In other words, the model matters more than the harness, and the harness effect looks like a fresh draw of the same noise.
Earlier comparisons, the authors argue, reported gaps above 20 points largely because they relied on a single run. Pure reruns of an identical setup can vary by up to six points on SWE-bench Verified, so a vendor’s advertised 2.4-point advantage falls inside that range.
The one measurable gap
The only statistically significant harness effect in the study is a deficit for OpenCode on the large task pool, where it trails the others by up to nine points. About half of that gap traces to an output-limit behavior: when a model hits the limit mid-step, Claude Code prompts it to continue, while OpenCode lets the step end, which usually ends the attempt. The authors report that copying a continuation prompt into OpenCode did not close the gap, because the same task rarely triggers the same failure twice. That detail is a reminder that harness bugs and defaults can matter even when sophistication does not.
Why sample size limits every comparison
The paper also quantifies what small benchmarks can detect. With 45 tasks, even a 13-point gap shows up only about half the time; with 447 tasks, gaps of roughly five points become reliably detectable. That means many advertised improvements, especially those of two or three points, sit below what the test can resolve. For readers who compare agents, the practical rule is to ask for multiple runs, confidence intervals and the number of tasks, and to treat any difference smaller than the rerun noise as unproven. Our guide to evaluating AI agents covers a broader checklist, and our piece on reward hacking explains why benchmark scores can be gamed.
Limits of the study
Several caveats apply. The work is a preprint with two authors and has not been through peer review. It covers one benchmark, SWE-bench Verified, whose tasks come from popular Python projects, so results may not transfer to other languages or to long, multi-day engineering work. Most models tested are open-weight or preview models; only one proprietary reference was run, in a single harness. Harness versions change quickly, so a result for a shipped version may not hold for the next release. Finally, a harness might still buy things that pass rates do not capture, such as safer defaults, better permission prompts or a nicer workflow.
What to watch next
- Peer review and replication of the preprint, including runs with other harnesses and newer models.
- Whether SWE-bench maintainers or leaderboard operators start requiring repeated runs and reporting noise ranges.
- Vendors publishing token cost per solved task alongside pass rates.
- Harness releases that cut token use while keeping pass rates, which the study suggests is the real opportunity.
- Follow-up work on multi-language and long-horizon benchmarks.
Key takeaways
- Holding the model fixed, three very different harnesses were statistically equivalent within about five points on 447 SWE-bench Verified tasks.
- Rerunning an identical setup moves scores about as much as swapping harnesses, so single-run gaps are weak evidence.
- Heavier harnesses cost two to three times as many tokens without a measurable gain in pass rate.
- The model, not the scaffold, drove the largest task-level differences.
- The study is a preprint and not peer reviewed; treat it as strong evidence about noise, not the final word on harness design.
Frequently asked questions
What is a coding-agent harness?
It is the system prompt, tools and context management that let a language model work in a code repository. Claude Code, OpenCode and mini-SWE-agent are examples.
Does this mean harness engineering is pointless?
No. The study finds that pass-rate differences were small on one benchmark, not that harnesses never matter. OpenCode’s measurable deficit shows that defaults and bugs can hurt, and harnesses may offer safety and usability benefits that scores do not capture.
Has the paper been peer reviewed?
No. It is an arXiv preprint (2610.04433), and we have not seen a peer-reviewed version.
