Wink Pings

Same model, pass rate from 40% to 80%—the only thing in between is a harness

Same model, same prompt: among 20 tasks, the number that pass directly goes from 8 to 16. The difference isn't the model; it's the control system around it (the harness). This breakdown comes from Trackmind, covering seven common failures and corresponding fixes, plus a checklist.

Recently, an article on X (formerly Twitter) was widely shared. The author, @0xTrackmind, says that when an AI agent performs poorly, the root cause is often the system around it; the model itself is usually not the main problem. This system now has a name: harness.

The tweet claims this is the same setup Anthropic uses to let a single agent work continuously for hours and self-check before saying it's done. One loop, three verification methods, one file you can copy directly, all on one page.

Screenshot of the Trackmind tweet

The article uses an experiment to prove this point: the same model, Claude Opus 5.5, the same prompt, with 20 tasks run twice. Group A: put the prompt directly into the repository with no other setup. Group B: the prompt stays the same, but a harness built over a weekend is added. Results: 8 tasks passed directly in Group A, versus 16 in Group B. Infinite retry loops dropped from 5 to 0, cost per task dropped from about $4.1 to $1.6, and human monitoring time dropped from two hours per day to 25 minutes.

LangChain data is similar: a coding agent's score on Terminal Bench 2.0 rose from 52.8% to 66.5%, with no model change—only the outer system changed. OpenAI has a similar report: three engineers drove about 1,500 PRs, with every line of code written by Codex. When progress was slow, they diagnosed the root cause as an environment definition that wasn't clear enough; the model itself wasn't weak.

The author gives a formula: Agent = Model + Harness. The model handles reasoning; the harness handles everything else: what it can see, which tools it can use, what it remembers between sessions, what counts as done, what it is forbidden to do, and what to do when something goes wrong. Put a model in a chat box and it only answers questions; put it in a repository with a terminal, tests, permissions, and review, and it can do real work.

The article focuses on breaking down seven failure scenarios and their corresponding fixes:

  1. It does something else: write the task contract first. Goal, constraints, deliverables, and done-when conditions. Add a final line: 'If you get stuck, stop, write your attempts to STATE.md, and do not guess.'
  2. It gets lost: give it a map, not a manual. An AGENTS.md of under 40 lines that clearly marks where the architecture, tests, product rules, and security docs are, so it can open them as needed.
  3. It says it's done but isn't: make evidence the acceptance criterion. Use check.sh to run type checks, lint, and tests; only when it outputs GATE: PASS does it count as done. Also have an independent verifier whose job is specifically to find reasons to reject it.
  4. It does things it wasn't approved to do: keep permissions outside the model. Tier them by reversibility: reversible actions are allowed, externally visible ones require approval, irreversible ones are blocked.
  5. It keeps retrying the same thing: categorize failures and set a retry limit. If the same error occurs twice in a row, stop, write the attempts to STATE.md, and escalate to a human.
  6. It forgets the decision it just made: store state outside the conversation. STATE.md records the current step, decisions, artifacts, risks, and next step.
  7. It makes the same mistake twice: turn lessons into infrastructure. Rules written into the prompt are less effective than rules written into lint checks. Once an error happens, make it part of what the environment enforces automatically.

The author warns not to start with a heavy platform. Start at Level 1: a task contract, a map, and a few tools. Every layer you add should correspond to a failure that has actually happened. Some parts of a harness are meant to compensate for current model weaknesses; once the model gets stronger, remember to cut them out.

The article ends with a checklist, suitable to go through before giving an agent real permissions. The full version is in the original article: https://x.com/i/article/2113375170392203264

发布时间: 2026-11-21 07:06