Wink Pings

All Tests Green, Bug Still There: One of the Most Expensive Failures for AI Coding Agents

Agents like Claude Code and Codex can pass tests and generate polished completion reports while never touching the original problem. kvro's Agent Reliability Engineering changes the definition of done from the model's self-report to verifiable evidence, budgets, and recovery boundaries.

This 27-second video is the demo from the source material.

You give Claude Code a task to fix checkout and let it run overnight. In the morning the terminal is green: 42 unit tests pass, type checking passes, the build passes, and the completion report is neatly written. Then a user submits an order with a certain payment method, and checkout still fails.

kvro calls this missing layer on X Agent Reliability Engineering. His assessment: an agent can generate 2,000 lines of code, run 40 tests, and produce a perfect summary while the original bug remains. More expensive still, it may spend another $20 fixing a problem that was never precisely defined.

This kind of failure has little to do with model strength or prompt quality. The core issue is the gap between 'producing an artifact that looks correct' and 'proving the intended result happened.'

The Green-Test Illusion

Tests passing only means tests passing. The original article calls it the Green-Test Illusion. The agent changes components, updates unit tests, and gets tests passing, but does not reproduce the original failure. Tests verify the behavior they were written to check. Whether the user's problem is solved requires a definition of done beyond the agent's own final answer.

Replace Prompts with Outcome Contracts

Prompts say 'what to try.' Outcome contracts say 'what conditions must be true before the system is allowed to report completion.' The examples in the original article include: reproduce the original failure, checkout e2e passes, no payment regressions, and the app builds; constraints include at most 3 attempts, at most $2, and protecting infra/production and .env; completion conditions require evidence, a summary of changed files, and human approval for deployment. The task changes from 'convince you I'm done' to 'produce evidence that satisfies the contract.' If the checkout test was not run, it cannot be marked verified. If a test fails, the task stays open. If the budget runs out, stop and escalate.

Evidence Engine and Five Components

A common mistake is having a second large model evaluate the first large model. Subjective review is fine, but it cannot replace execution evidence. The strongest checks for coding tasks are often highly deterministic: did the command actually run? Is the exit code correct? Did the failing test fail before the fix? Did it pass after the fix? Did the browser complete the user path? Were protected files changed? The original article provides a minimal Python verification layer that uses subprocess to execute commands and records passed, exit_code, stdout, and stderr. The key point is that success becomes a recorded observation. A sentence generated by the model no longer counts. The model proposes, the environment measures, and the controller decides.

The architecture does not have to be 30 agents chatting with one another. Five components are enough: a Planner turns the task into a bounded execution plan; a Worker makes code or configuration changes; a Verifier runs tests, browser checks, and acceptance conditions; a Recovery Controller decides whether another attempt is worth it; and an Evidence Ledger records what happened, what passed, what failed, and what remains unknown. The Planner and Worker can use the same model. The Verifier should use independent checks as much as possible. The controller enforces budgets in code; you cannot rely only on a prompt asking the model to 'respect the budget.' The Ledger must survive context resets.

The Recovery Loop Must Know When to Stop

Unbounded agent loops are dangerous because activity looks like progress. An agent may repeatedly edit the same file, rerun the same failing test, and burn tokens without discovering anything new. The original article gives three rules: every retry must have a reason; every attempt must produce new evidence, otherwise stop; and budget limits must be enforced outside the model. A simplified controller records a failure fingerprint, namely the exit code and error. If the fingerprint repeats, it returns STALLED. The best autonomous agents are not ranked by runtime. They stop when they know the next attempt has no measurable value.

Cost by Verification Results

AI agent economics is often described as cost per million tokens, but that is only part of the picture. A more useful metric is Cost per verified outcome. The original article uses hypothetical data: Agent A has a total cost of $30 and 55 verified outcomes, or $0.55 each; Agent B has a total cost of $42 and 84 verified outcomes, or $0.50 each. B is more expensive per attempt but cheaper per verified outcome. Optimizing only token cost is misleading. A cheap model that repeatedly produces unverified changes may be more expensive than a stronger model inside a good verification system. The economic unit of AI engineering is the accepted result.

Risk Tiers and Persistent Memory

Not all tasks should have the same approval process. Editing documentation is not the same as deleting production records. Low risk can be automatically verified; medium risk requires tests plus review; high risk, such as payment logic, requires independent verification plus approval. The key is to enforce policy through permissions and execution boundaries. A system prompt saying 'be careful' is not an authorization mechanism. A task that does not need production credentials should not get them. Deployment gates cannot depend on the model claiming to be confident. Autonomy should increase with evidence and reversibility, not with confidence.

Agents also lose information. When a session ends or context is compressed, important information may disappear. Persistent task records solve another problem: the next run continues from actual observations, and humans can understand why the agent stopped. Record the task, status, number of attempts, whether it is verified, results of each check, next step, and blocking reason. Memory stores decisions, evidence, and unresolved questions, not every token the model has generated.

What the Dashboard Should Show

Most AI dashboards show token usage, number of completed tasks, number of active agents, and model latency. That is not enough. A reliable engineering dashboard should also track: verified completion rate, false completion rate, cost per verified outcome, recovery efficiency, human intervention rate, and time to verified resolution. A system with 500 completed tasks but only 200 verified outcomes is not necessarily more efficient than one with 300 completions and 280 verified outcomes. Metrics make different agent architectures comparable without relying on demo effects.

A Practical Starting Point

Start with one repository and one type of repetitive task. Create three files: AGENTS.md describes the repository, test commands, architecture rules, and required verification behavior; tasks/checkout.yaml defines acceptance conditions, limits, and protected resources; evidence/checkout.json stores observations. The instructions to the agent should require it to first inspect the code, reproduce the failure, and record the failing verification command; after making changes, run the original reproduction test and relevant regression tests, record command results and changed files, and mark unverified acceptance conditions. Do not report VERIFIED unless all required checks pass and evidence is recorded. If the same failure repeats with no new evidence, stop and report STALLED. If authorization is needed, request approval and do not bypass protection boundaries. Prompts establish workflow, but an external controller must still enforce permissions, budgets, and completion rules. Prompts are instructions; guardrails are implemented boundaries.

In the discussion, one commenter said an agent passed all the checks he wrote and still shipped a bug his tests had not considered. He now distrusts the test suite first, and then distrusts the model. Another commenter mentioned PDF extraction: after switching JSON to CSV, parsing errors disappeared, but rows were still missing. The parser was happy; extraction was not complete. Someone else said that having an agent write tests to judge itself is like grading its own homework. At least one check it cannot modify is needed.

The last line of the original article is worth putting here: stop measuring how much code AI writes and start measuring how many problems it can prove it has solved. Better models matter, and better prompts also matter. Engineering advantage will accumulate in the system around the model.

Source material and further reading:

发布时间: 2026-11-12 02:27