研究发现复杂编码智能体框架只增加成本不提升效果
Fancy harnesses make coding agents more expensive but not better, study finds
AI 导读
一项研究对比 Claude Code、OpenCode 和 mini-SWE-agent 三个编码智能体框架，发现框架切换带来的成绩变化与同一配置重跑一次的波动相当。
摘要与补充说明
研究用 Qwen3.6、Qwen3.8、Deepseek-V4-Flash、GLM-5.3-Flash 和 HY4-Preview 五个模型在 SWE-bench Verified 的 45 个最难任务及 447 个任务池上测试，同一配置重跑成绩可差 6 分，换框架时任务级结果翻转率同为 13%，换模型则升至 22%。唯一显著差异是 OpenCode 在大任务池落后最多 9 分，约一半源于输出超限后不续写。
正文· AI 翻译
译文尚不完整，完整内容请切换到原文。
Earlier comparisons were measuring noise
Previous studies report gaps of more than 20 points between harnesses, but those numbers almost always come from a single run, according to the authors. Pure reruns of the same setup already vary by up to six points on SWE-bench Verified. Zhipu AI, for example, advertises its own harness with a 2.4-point lead over Claude Code, a margin that falls well within that noise range. Harnesses are also often wired into the benchmark differently, making comparisons even less reliable.
Liu and Han ran five models through three harnesses in their shipped versions: Claude Code, OpenCode, and mini-SWE-agent. mini-SWE-agent gives the model only a Bash tool and skips context management. Claude Code and OpenCode offer specialized file and search tools, long system prompts, and automatic history compression.
The models were Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview. Claude Opus 5 ran as a reference only in Claude Code. Tests used the 45 hardest tasks from SWE-bench Verified plus a pool of 447 more. Several setups ran twice to measure noise.
All runs happened offline. In early tests with web access, 71 percent of Deepseek and Qwen model attempts pulled the correct fix straight from GitHub instead of solving the task themselves. Claude models almost never did this. The researchers discarded those results and don't claim any maker trained on the test data, but the pattern was strong enough that they cut off web access for all remaining runs.
Swapping harnesses moves results about as much as a rerun
On the larger pool of 447 tasks, Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, sit less than two points apart for both tested models. Within a five-point band, they're statistically equivalent.
The researchers expected the biggest harness effect on the 45 hardest tasks. Instead, harnesses differ by two to five solved tasks per model, and a simple rerun of the same setup shifts the count by up to three. Sometimes mini-SWE-agent leads, sometimes Claude Code does.
The flip rates tell the same story. Between two reruns, the median outcome changes on 13 percent of tasks. Between two different harnesses, it's also 13 percent. Swapping models pushes that to 22 percent. Which tasks a given harness wins changes from run to run, matching a coin flip. The harness gap behaves like a "fresh draw of the same noise," the authors write.
These small samples also limit what any comparison can detect. With 45 tasks, even a 13-point gap shows up only half the time. With 447 tasks, gaps of about five points become reliably detectable. Zhipu's claimed 2.4-point advantage falls below both thresholds, and so do many other advertised improvements.
All harnesses perform the same, except OpenCode falls behind
唯一可测量的 harness 效应是 OpenCode 的稳定劣势。在大型任务池上，它落后另外两者最多九个百分点。其中约一半的差距可追溯到输出上限。当模型在推理过程中于某一步中途超出该上限时，Claude Code 会提示它继续。OpenCode 则让该步骤就此结束，而这次尝试通常会失败。
给 OpenCode 加上 Claude Code 风格的续写提示并没有帮助，因为同一个任务在重试时很少触发同样的溢出。至于差距的其余部分，作者没有解释，只有一个观察：在 OpenCode 下，模型停止工作的时间要早得多。在本地运行的模型上，OpenCode 的时间限制也比 Claude Code 严格得多，作者将其标记为一个未受控变量。
mini-SWE-agent 因为缺乏上下文管理而撞上上下文窗口，主要出现在其他 harness 也失败的任务上。OpenCode 的搜索工具在首次使用时会尝试下载一个二进制文件，离线时就会出错。
系统提示与工具描述决定了账单
在相同模型、相同任务和相同价目表下，单个任务的成本因 harness 不同最高可相差三倍。工具输出几乎无法解释这一差距。三个 harness 每次调用返回的文本量相近。
真正的差异来自系统提示和工具描述。它们会在每一步重新发送，Claude Code 约为 16,600 个 token，OpenCode 约为 7,000 个，mini-SWE-agent 则不到 1,000 个。根据研究人员的逆向工程，Claude Code 还会将模型的推理输出保留在对话历史中。
在首次调用之后，所有 harness 的上下文增长速度相近。系统提示的大小决定了基线成本，而步数会将其成倍放大。哪个因素占主导取决于模型：使用 GLM-5.3-Flash 时，Claude Code 的成本几乎是 mini-SWE-agent 的两倍。使用 Deepseek-V4-Flash 时，mini-SWE-agent 需要多出许多步，最终两者持平。缓存 token 的更低价格会缩小差距，但不会改变排名。
没有主场优势，围绕思考的未解问题
Deepseek 自家的 harness 搭配自家模型时，得分并不比其他 harness 更好，但成本最高。即使研究人员从 Claude Code 中移除了工具，Claude Opus 5 仍给出完全相同的结果。一年前的版本以不到一半的成本解决了相近数量的任务，因为当时 Extended Thinking 并未默认开启。
思考是 harness 选择真正改变结果的唯一情形。在不开启思考时，使用 Qwen3.6 的 mini-SWE-agent 下降十一个百分点，而 Claude Code 和 OpenCode 的波动在噪声范围内。HY4-Preview 在 Claude Code 中不开启思考时也下降十一个百分点。智能体对推理的依赖程度似乎与具体的模型-harness 组合有关。作者承认他们的测试网格太小，无法解释原因。
该研究主要依赖一个基准测试。在 Terminal-Bench 2.1 上进行的小规模交叉验证显示了相同的模式。闭源顶级模型 Opus 5 仅在 Claude Code 中运行。作者总结道，一个 harness 换来的是“更少的任务失败方式，以及一份账单，而非通过率”。他们建议在每次 harness 对比时，同时发布相同设置的复跑结果。这是判断测得的差异中有多少只是噪声的唯一方法。数据和分析脚本可在 GitHub 上获取。
在其他研究中，harness 中投入更多精力也很少带来回报。在一项研究中，一个简单的 Markdown 文件胜过了面向 AI 编码代理的复杂技能系统。此前的一项研究已经表明，框架的选择可能使同一模型的成本几乎增至三倍。因此，harness 设计受到了大量关注。Nvidia 最近通过更好的代理引导，将其编码代理的 token 成本削减了近一半。