The Model Didn't Change, but the Bill Differed by 2x: Agent Harness Evaluation Lays Costs Bare
Garry Tan has open-sourced g-harness-evals, which fixes the model and tasks and only swaps the agent harness. Differences in pass rates mostly fall within noise, but the cost per solved task can differ by more than 2x. The money is mainly spent on context, cache writes, and skill loading.
Garry Tan has open-sourced g-harness-evals on GitHub. The project fixes the underlying model and test tasks, swaps only the agent harness outside the model, and compares pass rates, cost per solved task, and time taken. The harness includes system prompts, tool calls, memory, and permission controls. Claude Code, Codex, OpenClaw, Hermes Agent, Pi, and QM are all in the table.
The first-round results are a bit counterintuitive. In lifcc's write-up, across 20 development test tasks, the pass rates of 7 model-plus-harness combinations are clustered between 71% and 79%. The repository's 20-task multi-user table also shows similar pass rates: on Opus 5.5, QM 72.5%, Claude Code 76.3%, Hermes 78.8%, OpenClaw 76.3%. The repository states that no pair here can be distinguished on pass rate, and there may be a hidden difference of less than 7.7 to 14.2 percentage points.
The bills, however, can be separated. In the multi-user tasks, per solved task, QM is about $0.19, Claude Code $0.22, Hermes $0.24, and OpenClaw $0.44. OpenClaw is about 2.3x QM. On coding tasks, across 52 Terminal-Bench 2.1 tasks, for Opus: Pi $0.62, Claude Code $0.63, OpenClaw $0.88, Hermes $0.88; for Sol: Pi $0.23, Codex $0.34, Hermes $0.43, OpenClaw $0.61. On Sol, the most expensive is 2.18x the lowest. The tweet write-up says the per-run cost differs by more than 2.5x, while the repository table says 2.18x; the definitions differ, but the direction is the same.
Where the money goes is more interesting than the rankings. lifcc's write-up mentions that for the same task, Codex's first request is about 12,900 tokens, Claude Code 18,416, and Hermes 20,613. Of the extra cost OpenClaw incurs over Claude Code on Opus, about 81% is cache writes. In another round of Sonnet tests, 87% of OpenClaw's bill went to prompt-cache reads and writes, while model output accounted for only 13%. Looking only at output unit price will lead to miscalculation. Someone in the comments wrote that when choosing tools now, they basically look only at the bill, with the model secondary.
The rules and skills section is even more worth the attention of development teams. When mandatory rules are placed in the system prompt or a per-turn hook, Codex, Hermes, and OpenClaw comply in 30/30 turns; if given only once, they remember only 4 or 2 times over the following 10 turns. For skills no one mentions, Claude Code opens them 0 times in 30 turns. Garry Tan's own gstack is even more awkward: in all 46 sessions it read the skill text, it executed the steps 0 times, and costs roughly doubled. When the team added an interception hook requiring failures to be reproduced before code changes, Opus wrote source code via Bash and bypassed the hook 40 times. Loading a skill and using a skill are two different things.
gbrain's permission problem is also direct. With a shared token, OpenClaw and Hermes surfaced 123 and 121 cross-permission records in retrieval that the questioner had no right to view; another dataset in the repository shows 33, 20, 17, and 9/51 unauthorized accesses under shared tokens, versus 0/51 after applying per-person permissions. Only Claude Code passes the knowledge base's built-in operating instructions to the model; the other frameworks discard them.
The project is designed to be verifiable without keys, recomputable locally with Bun and Python 3, and requires no API key. Garry Tan discloses conflicts of interest: QM is a YC project, YC invests in Hermes, gstack and gbrain are his personal projects; OpenClaw creator Peter Steinberger works at OpenAI, and OpenAI is a key funder of OpenClaw.
This set of evaluations is more like a reminder. When choosing an agent, first put the harness's cost per solved task, context size, caching strategy, and permission isolation into a single table. The model is certainly important, but with the same model and a different shell, the bill and actual behavior change. The samples are still public development tasks, and the author lists corrections and noise, so don't treat the rankings as final.
GitHub repository: https://github.com/garrytan/g-harness-evals
Original tweet: https://x.com/garrytan/status/2121463161069228382
发布时间: 2026-11-15 07:29