Inside Anthropic, No More Rubber-Stamping Each Other: Code Review Goes to AI, Rollbacks Drop 40%
More than half a year ago, Anthropic relied on team members rubber-stamping each other's PRs to let them through. Now they have AI agents take over most code review; PR merges have doubled, and rollbacks are down 40%. This article breaks down how they did it, what pitfalls they hit, and which steps ordinary teams can copy.
More than half a year ago, a saying circulated inside Anthropic: “Clark, you stamp my PR, and I'll stamp yours.”
Claude writes code faster and faster, changes waiting for human review form long queues, so people just let each other through: see who submitted it, glance at it, approve. Morgan Lunt, a product manager on the Claude Code team, called this “kabuki theater”: the form of review was still there, but the substance was gone.
A PR is the process by which programmers submit code changes, have them checked, and merge them into the main codebase. About nine months ago, models became strong enough to complete a feature independently; people started handing tasks directly to Claude, output surged, and review became the slowest link. Lunt cited a Harvard study: after companies introduced AI coding agents, the average wait time for PR review increased by 56%.
Around July this year, Boris Cherny, the creator of Claude Code, went to Lunt's team and asked them to solve Anthropic's code review problem. The team set its own goal: when 90% of PRs are merged, no one besides the person who got Claude to do the work should need to be brought in. At the same time, there was a bottom line: rollbacks, emergency fixes, and production incidents must not increase because of this.
Results: the proportion of PRs in which substantive issues were found rose from 16% to 54%, the number of merged PRs doubled, and PRs needing rollback fell 40%.
Seven Bots Are Worse Than One
At first, engineers each wrote review bots, encoding what they cared about when reviewing code. A PR would see seven bots, their opinions conflicted, leading to 50, 100, even 200 back-and-forth commits, diverging from what the author originally intended to do.
Later they merged the concerns into one main review agent, which deduplicates and organizes them internally. The finalized process: first dispatch a group of issue-finding agents, then a group of “nitpicking” verification agents specifically to challenge the former's findings; once a finding is confirmed as a real issue, it is ranked by severity and posted as a comment on the PR. Major issues block merging, minor issues do not, and pre-existing issues that existed before the change are listed separately.
To make review finish in a reasonable number of rounds, Lunt spent three weeks. If an agent is asked to find problems, it will always find problems; if asked to fix them, it will always go and fix them. For a while, a PR would go back and forth overnight, accumulating dozens or hundreds of commits that no one could understand. The fix was to have the agent report all findings at once and, after they were fixed, stop digging deeper. Now more than 90% of PRs converge within three rounds of review and fix.
They also tried giving the review agent more tools, more context, and letting it actually run code; most of this brought no improvement and instead increased cost and complexity. At present, the best cost-performance internally is Opus 5.5 at medium reasoning effort.
Exam Questions Come from Our Own Incidents
How do you judge whether a review agent is good? They compiled all production incidents in Anthropic's history, plus the PRs identified as causing incidents, into an evaluation set. Each question records the repository version at that point in time, the PR number, the files and line numbers where the problem occurred, and the correct answer. The review agent is placed back at that point in time to review, and the test is whether it can find that line.
They did not use synthetic data with artificially planted bugs; all exams are real cases that actually went wrong in their own code. They measure two numbers: recall, how many of the issues in the ground truth were found; precision, how many of the reported issues were truly worth reporting. The method is trial and error: change one thing, keep it if the score improves, discard it if it gets worse.
Six Layers of Defense and Risk Tiering
The whole process has six layers: in the session where Claude writes code, it first runs tests itself and dispatches subagents to recheck; after the PR is submitted, hosted review performs a deep review; then a security review; then regular automated checks and tests; only PRs that really need it get human approval; finally, merge.
They call this “Swiss cheese” style defense: every layer has holes, but stacked together it is hard to slip through.
After review there is also risk tiering: a bot decides how many approvals are needed based on which files and how much the PR changed. Docs-only changes pass directly; for ordinary changes, the signature of the person who got Claude to do the work is enough; for high-risk areas such as billing and auto-updates, a second person's signature is required. Risk is assigned based on the change itself, not the author.
There is also a fallback layer: most Anthropic products are released through the web and apps they control; new features are off by default, first enabled for internal testers, then gradually rolled out. If something goes wrong, it can be turned off for everyone within minutes without changing code.
After review finds issues, fixes are automated as well. Internally, some people use Claude Tag, while others write their own “PR-watching” skills, letting the Claude session that wrote the PR periodically check GitHub for review comments. If fixes still wait for humans, speedup is impossible.
Who Is Responsible for AI-Written Code?
Some ask: AI writes, AI reviews, humans are left only to click approve, so who is responsible if something goes wrong? The team's principle is “whoever causes this change to happen is responsible.” The person who got Claude to do the work bears the same responsibility as the engineer who wrote the code by hand. Review agents only handle the middle layer: whether the change is written correctly and safely; the top-level question of whether this feature should be built at all remains with humans.
Which Steps Ordinary Teams Can Copy
Lunt suggests: first build an evaluation set from your own incident records. Most companies have incident tracking systems; have Claude extract the PR numbers that were identified as causing incidents, organize them in the format above, and it can be done in a few days. Another simpler method: take PRs that were previously reviewed by humans, have automated review go back to that point in time and review them again, and see how many of the issues humans raised back then it can find.
When tuning review rules, you can put a REVIEW.md in the repository root, written specifically for the review agents to read. Auto-generated code does not need to be reviewed, code style is already handled by tools, minor issues should be reported at most a few, and specify the format in which findings should be written. CLAUDE.md continues to contain the rules that everyone and every agent participating in the repository must follow. An online commenter added a detail: the hosted Code Review service reads both the REVIEW and CLAUDE files; the local /code-review command only reads CLAUDE, not REVIEW.
This case has also been discussed quite a bit in Chinese tech circles. Some said “rubber-stamping each other is too real; the faster you write, the more perfunctory the review becomes”; others reminded that low-risk PRs can be merged without human approval, but only if there is a fallback of staged rollout and fast rollback behind it; billing and permissions cannot just copy auto-approval; others questioned the leap from 16% to 54%, saying AI review is more likely to count style issues as substantive issues, and one should look more at how many of them would change production behavior. But most agree with one practice: using historical incidents as exam questions is more reliable than tuning prompts by feel.
Lunt compares working on the Claude Code team to building sandcastles by the sea: the auxiliary structures you build get washed away by the capabilities of the next-generation model. He would not be surprised if code review becomes a completely solved problem before long. In the live poll at the beginning of the talk, 77% of the audience said review is the most painful part of the development workflow.
Information comes from the Claude Code team's official blog (https://claude.dev/blog/solving-code-review-at-anthropic/), relayed and shared by Baoyu (@dotey).
发布时间: 2026-11-20 14:56