20 Tasks: The Gap Between 8 and 16 Is Not in the Model

Boris Cherny recently released a research note on Opus 5.5 and agent harnesses. Trackmind then expanded it into an actionable guide with a very direct title: Unlock Opus 5.5: The Harness Guide for Agents That Actually Ship.

First, some context. Harness refers to the layer of system outside the model: what the model can see, which tools it can call, what it remembers between sessions, what counts as done, which actions are forbidden, and how it stops after a failure. The model is responsible for reasoning and proposing the next step; the harness is responsible for turning that next step into an action that is executable, checkable, and rollbackable. Put the model in a chat box, and it answers questions; put the same model in a repository with a terminal, tests, project notes, permissions, and a reviewer, and it might actually get work done.

Trackmind recorded a three-week cycle: the agent says it is done, but the tests are red; it changes config files that were never mentioned; an overnight task retries a failing API 41 times until the budget cap stops it. Most people at this point rewrite the prompt, switch models, or stuff in more documentation. Trackmind's judgment is that the prompt was never the main problem. Opus 5.5's capabilities already exceed what many setups allow it to demonstrate.

Public data points to the same layer. LangChain raised a coding agent's score on Terminal Bench 2.0 from 52.8% to 66.5%, moving from outside the top 30 into the top 5. Terminal Bench 2.0 is a benchmark for coding agents. The model did not change; what changed were the prompts, tools, and middleware. OpenAI described an experiment: three engineers pushed about 1,500 PRs into a repository, with every line of code written by Codex. Early progress was slow, and the diagnosis landed on insufficient environment description; model strength was not on the list of problems. The engineers' work shifted from writing code to designing the environment, stating intent, and building feedback loops.

Trackmind ran a smaller controlled experiment of its own. 20 real tasks: bug fixes, small features, and a few refactors. The same Opus 5.5, the same base prompt, the same machine, run twice. Group A just dropped the prompt into the repository. Group B added a harness built over a weekend: a short contract for each task with a "done when"; a 40-line map file replacing the previous pile of documentation; a test gate that prevents anything from being called done if it does not pass; a retry cap of 3; and a rules file spelling out what the agent may and may not touch. Results: in Group A, 8 of the 20 tasks needed no human takeover; in Group B, 16 did; runaway retries were 5 versus 0; average cost per accepted task was about $4.10 versus $1.60; and daily time spent watching the agent dropped from about 2 hours to 25 minutes. The model did not change, and the prompt wording did not move either.

This is not mysterious. Trackmind breaks common failures into seven categories, each mapped to a fix in the harness. If the task goes off track, write a contract first, especially a "done when": tests pass, analytics events fire, and both desktop and mobile flows pass. If the agent gets lost in the codebase, give it a map, not a manual: a root-level AGENTS.md or CLAUDE.md under 40 lines, stating only where to find architecture, tests, product rules, and security requirements, with detailed docs opened on demand. If it says it is done but is not, turn evidence into the completion gate: type checks, lint (code checking), tests, and end-to-end scripts; if they do not pass, it is not done; make the reviewer role independent, and write the task as finding reasons to reject rather than asking whether it looks good. If it does things without approval, put permissions outside the model: allow reversible operations, require approval for externally visible actions, and shut off the channel entirely for deleting data, rotating credentials, or publishing globally. If it retries repeatedly, classify the failure first and then set a cap: timeout, bad parameters, missing information, test failure, and permission denial each have their own handling; stop when the same failure appears twice. If it forgets decisions, put state outside the conversation: a STATE.md recording the current step, decisions and reasons, artifacts, risks, and next steps. If it makes the same mistake twice, turn the lesson into infrastructure: if you can lint it, do not just write a prompt; if you can test it, do not rely on memory.

Trackmind's guide also includes a reminder: some parts of the harness are compensating for today's model weaknesses; as models get stronger, they will become a burden and should be deleted regularly. This judgment is calmer than the idea that prompts can do everything.

For ordinary users, the most valuable takeaway is the layering: if the agent answers the wrong task, look at the contract first; if it answers fluently but is outdated, look at context first; if it says it is done without evidence, oversteps permissions, or forgets things, look at the harness first; if it gets stuck repeatedly, look at the loop first; if multiple agents collide, look at the coordination map first. Prompts only solve the first layer. A longer prompt is usually just tossing one more coin into a wishing well.

Original tweet: https://x.com/0xPontiuz/status/2113998765111808369
Trackmind guide: https://x.com/i/article/2113375170392203264