Wink Pings

At 4 A.M., the Agent Reports “All Done.” Reality: 11/40

Anthropic disclosed the complete methodology for building its own agent. The problem was never the model; it is the layer around the model called the harness. A 6-layer structure, 7 failure fixes, 1 prompt that packages everything, and a story about 40 endpoints at 4 a.m.

Your agent didn’t suddenly get dumber at 2 a.m. Its context was stuffed. It didn’t “decide” to skip tests. There was simply nothing in its world that made tests mandatory. Nor did it “hallucinate” that the task was done. It simply had no way to see that it wasn’t.

Anthropic recently disclosed how they build their own agent. The model was the same; they took apart the part around the model, called the harness.

Trackmind posted a long piece on X describing a typical scenario. The task was to migrate 40 endpoints. The model ran in a chat window, and the first 12 endpoints went smoothly. Then the context filled up with code changes and test output, and compaction happened. It forgot which endpoints were done, forgot which naming convention it had chosen. Endpoint 9 was redone using a different naming convention. At 4 a.m., it reported: all 40 endpoints migrated successfully. Reality: 11/40, two naming schemes, tests never run.

The same model, run inside a harness. The task becomes a contract. Progress is stored in files, not in the conversation. Every endpoint has a test that must go from red to green. Retry count is capped. Database configuration is read-only. Result: 39/40 green, 1 escalated to a human with a clear reason.

Anthropic described this failure mode in its paper: “At the start of each new session, it knows nothing about what came before.” “The next agent instance looks around, sees progress, and declares the task complete.” Their fix was not a smarter model, but a progress file, a structured feature list, git checkpoints, and real end-to-end tests.

The model didn’t change. The harness changed. The result was different.

They broke down their method for building an agent into 6 layers:

  1. Tools: 6, not 20. Each does one thing. Every error tells the agent what to try next.
  2. Context: a very short CLAUDE.md instruction file. Loaded when needed; never dump the entire repo in the first turn.
  3. Permissions: reads and tests are auto-approved. Deployment and deletion must ask a human.
  4. Hooks: don’t rely on the model to remember to run code checks. After every edit, the hook runs automatically.
  5. Verification: the agent never grades its own homework. Use a reviewer who was not previously involved to read the code changes and run the tests.
  6. Evaluation: every harness change is scored before shipping.

image

These 6 layers are packaged into one prompt. Paste it into Claude Code, and it helps you build the entire harness. 6 layers, 1 prompt, 0 infinite loops.

Trackmind broke the 7 most common agent failures down one by one into “sounds like a model problem, actually the harness is missing a piece”:

  1. It does a different task, not the one you asked for. Missing contract. Write the task as a contract: goal, scope, prohibitions, completion conditions. done_when is the most important line; if you can’t write it, the agent will never complete the task, it will only stop.
  2. It ignores project rules. Missing map. A very short AGENTS.md that gives only directories and pointers. Rules live next to the code they govern and are read only when needed.
  3. It chooses the wrong tool. Missing tool boundaries. One tool does one thing, has small output, has loud errors, and declares what it can touch.
  4. It forgets decisions it just made. Missing state. Write PROGRESS.md and add a git commit. Anything not written to a file didn’t happen.
  5. It says “it should all pass.” Missing sensors. Exit codes, screenshots, contract tests. Only what can say “no” counts as evidence. A harness that treats “done” as evidence is itself a bug. Anthropic found the same issue: Claude is inherently a poor QA agent. The solution is to separate the builder and the judge, not to swap the model.
  6. It retries the same fix infinitely. Missing exit. Stop when the same error appears twice; stop when the budget is exceeded; escalate to a human. What it lacks is a fact, not another attempt.
  7. It touches what it shouldn’t. Missing walls. Permissions are infrastructure, not sentences in a prompt. Let the model propose any action; let the harness decide which ones actually execute. Dangerous actions always require human approval.

How do you tell whether it’s the model’s fault or the harness’s fault? Three questions: Can it see? Can it do it? Can it judge? If any answer is “no,” the harness has a bug. Put a smarter model into the same blind spot, and it is still a blind spot. There is a practical test: give a capable human engineer exactly the same things as the agent—same files, same tools, same context. If they fail too, then it is not a model problem.

You don’t need to build every layer on day one. Every layer should be paid for by a real failure. A one-off problem needs only a prompt. A 20-minute task needs a contract, a map, and state. An overnight run with write permissions, plus PRs, needs all of them.

Trackmind is not alone in this judgment. OpenAI, in codebases where agents write code, also used the same term: harness engineering. AGTP summarized the related discussion on X, analyzed 50+ posts and 5,500+ practitioner comments on r/AI_Agents, and found that real failures in production and the things hyped in demos are basically two different things. 7 major pain points emerged.

image

MIT CSAIL published a paper on November 20 titled Language Model Harnesses Can Be Compositional Generalizers. The paper argues a counterintuitive phenomenon: many coding agent harnesses repeatedly append history, tool output, and reasoning into the prompt. A continuously growing transcript is not a training distribution, so the longer an agent session gets, the dumber it may become. A good harness should keep every input it gives the model in a form the model is familiar with, even if the overall task is unfamiliar.

There were dissenting voices too. Matt Shumer said on October 31 that harness engineering is usually a mistake. His approach: use the harness that shipped with the model during training, and write good behavior prompts. Others added: don’t over-engineer; keep it simple.

The security perspective was present too. Kirk Borne mentioned a book on harness design patterns, covering security control planes, prompt injection, and memory poisoning. Brian Armstrong observed that many AI-written PRs still require substantial revisions, and the burden on human reviewers has not eased.

Someone validated the importance of tests firsthand. Andrey Kucherenko said he had an agent prove every refactor commit with tests; the agent eventually stopped in front of a 6,388-line module that had no tests at all. He said an agent that never stops is the one he trusts least.

The model will change next quarter. The harness is the thing you keep. Stop asking which model is smart enough; ask which environment makes it reliable. That is harness engineering.

发布时间: 2026-11-22 11:04