Wink Pings

Claude Wrote 90% of the Code—First Make It Show the Evidence

The Claude Code team turned verification into a system: end-to-end, property, and differential testing, plus one hard rule—code under review cannot modify the checks. A commenter's addition cuts deeper: tests written in the same pass mostly confirm only what the agent already believes.

The Claude Code team writes and reviews about 90% of its code with Claude. That figure comes from Addy Osmani's article on claude.dev. The question follows: who guarantees that this code actually runs? The team's answer is not 'trust the model' but 'make verification infrastructure.'

The verification loop determines how many Claudes you can run at once. Without it, you can only run one or two, because you have to inspect every diff yourself. With it, dozens or hundreds of agents working in parallel become meaningful. In internal tests, structured file-generation tasks plus an independent grader checking against a rubric raised task success rates by up to 10 points; the harder the problem, the bigger the gain.

Verification that used to be expensive—tests, benchmarks, specs, real end-to-end runs—can now be generated quickly by agents. Once generated, tests and benchmarks run repeatedly as software and no longer consume large amounts of tokens. The more important step is closing the loop: if an agent repeatedly gets stuck on a check—for example, user data showing up in logs—add a hook or write an explicit rule in CLAUDE.md so the same class of problem is less likely to happen again.

Three Kinds of Tests, Start with the Hardest to Fool

If you can choose only one, choose end-to-end. It describes a real user flow: add to cart, check out, reach the confirmation page. End-to-end is hard to fake because the whole feature either works or it does not; the completion criteria are also clear. Unit tests are faster and more stable, but they can all pass while the feature is still broken. This is common when neither the agent nor the tests understand the full task.

单元测试失败和端到端测试失败的对比

End-to-end tests can give an agent hands and eyes. Use a skill to start, drive, and inspect a real running system; the tools can be CDP, an emulator, or an API harness. A feature map then serves as a guide and scoring rubric, usually listing paths, shortcuts, selectors, and expected results from the user's perspective.

Property tests target boundaries that people easily miss and agents easily get wrong. First write a statement you believe cannot fail, then use a test library to generate thousands of inputs to try to disprove it. Common properties include: decoding after encoding should restore the original; an account balance cannot go negative; running a migration or retry twice has the same effect as running it once; and optimized code produces the same results as naive but correct code. The value of a property-testing library is that the boundaries it generates do not carry the coding agent's biases. Property tests are especially useful when specs drift or production behavior and documentation disagree.

Differential testing is suited to large-scale migrations. The old system and the new system receive the same inputs; if results differ, there is a bug or a breaking change. Inputs can be randomly generated or sampled from production traffic. You also need to normalize irrelevant factors such as whitespace and maintain an approved list of differences explaining where the new environment is supposed to differ.

How the Team Keeps Verification from Becoming Self-Deception

The Claude Code team treats running changes as more important than testing. iOS changes must run in a simulator, web changes use a Chrome extension, mobile uses iOS or Android MCP, desktop uses computer use, and backend runs the service locally. They write startup and run instructions as a verify skill, place it in the repository's .claude/skills/ directory, and keep it as close as possible to the relevant code. Before committing, Claude uses it for smoke tests. The skill itself must also be iterated on and stress-tested; it has to be fast and stable enough.

Handling failures is clear. Claude runs the checks and tries to fix them. If a check is still failing after three rounds, Claude stops and hands off to a human. Code owners write their own checks and tune them until they flag what the owner would flag. When a check pulls someone in, it should explain why, rather than making the reviewer guess from the diff.

They also maintain a 39-item reliability checklist. Each item has a trigger condition readable from the diff, a check, and the failure or PR behind it. For example, --resume loads an old transcript. When session storage or message types change, you need to ask whether the new field can be loaded safely, because the old transcript does not have it. Claude reads the diff and checks only items whose trigger conditions match—usually 3 to 10 items. No one needs to memorize every historical failure.

Review does not rely on piling on people. Adding a more expensive second reviewer is often worse than tuning the first reviewer. They use REVIEW.md to tell the reviewer what to pay attention to in this code. Each round of review and verification continues from the previous round's context and failures instead of rechecking from scratch; it is cheaper and can find more.

There is one hard rule: a change cannot modify the checks that judge it. Code Review reads REVIEW.md from the base or default branch, never from the PR's changes. The rule lives on a branch the change cannot touch. Review locks the commit; if someone pushes while review is in progress, the result is invalidated. Cheap checks run first. If an agent pushes without testing locally, it wastes shared review and CI time. The team requires Claude to run tests, builds, and the linter before opening a PR, and can enforce this with a hook or /goal.

New checks start in observe mode. Every commit is stamped with its verification method, including none, so you can query commits that were not verified. The commit gate has three levels: off, observe, and enforce. Observe mode only records commits that would have been blocked, for reviewing false positives. Only enforce mode actually blocks. The commit message can explain why something was not verified. After being blocked twice in one session, the next attempt is allowed, to avoid Claude getting stuck.

They also review the reviewers. They track changes that were rolled back after merge, changes maintainers rejected before merge, and changes matched by size that stayed clean. Evaluation reconstructs the repository state from historical failures and rollbacks, gives one-sentence answers, and scores only findings that correctly describe the bug. Merged and stable changes serve as a control group to prevent over-flagging. Recall is looked at separately for raw findings, post-verification findings, and final review findings, so the system does not lose bugs it has already found. Importance is also layered: must fix, should fix, minor issue. Code Review labels Important, Nit, and Pre-existing. Pre-existing means the reviewer should not make the author carry old bugs. When recall is tuned so high that further increases would mean overfitting the evaluation rather than actually catching bugs, stop.

Missed cases should become rules. One method is to extract patterns from human review comments and PR rollbacks. A problem qualifies as a candidate rule only if it appears in at least two PRs and can be identified from the diff. Three agents try to disprove it, and then a human reviews and approves it in a PR. Another method is to replay three bug finders against PRs associated with known failures. It counts as detectable only if at least two find it. If it is missed on a small diff, it is worth writing into REVIEW.md. Flaky tests are handled the same way. Claude cannot skip them, add sleeps, or retry until they pass. If the root cause cannot be found, delete the test and write down why. Rules also need cleanup. Test suites, verification steps, and .md files only grow, so they must be pruned regularly.

Commenters Add an Even Sharper Point

One commenter said the problem is who writes the checks. When an agent writes code and tests in the same pass, it mostly confirms what it already believes. Another commenter said the most trusted tests are the ones the agent is not allowed to edit; otherwise, its way to turn red green is to delete the assertions.

网友讨论截图

Someone described their verification loop: Claude says it is done, they ask 'Are you sure?', and Claude finds three bugs. The process sounds like a joke, but it shows that verification cannot rely only on an agent reporting completion. Another commenter said unit tests once all passed because the tests and the agent's bug agreed with each other; the polite lie stopped only when the agent used the app like a normal user.

What gets handed to a human after three failed rounds is also critical. Three failed diffs are a mess. A write-up stating which check failed, what has already been tried, and why it is still stuck is what lets a human take over.

On the rule that checks cannot be modified, one commenter said this is the most important point. Otherwise, the verification loop becomes something the agent optimizes for. Keep the rules on a protected branch and enforce them with CI; this can be done outside Claude Code as well.

总结截图

Others pointed out the other side: one commenter said they deleted 75% of their tests over the weekend. More test suite is not always better; bad tests train agents to rerun rather than fix the code. Slow tests make agents reduce batch checks, making problems harder to locate.

The verification layer should be treated like any other system. The power to check must be independent. Score checks with the bugs you actually care about. If something is missed, write a rule. Rules need pruning. Once this is done, the work shifts from reading every diff to defining what done means and looking at evidence. In your current verification loop, which step can Claude change? That question is more practical than 'Can it write code?'

发布时间: 2026-11-17 10:28