Anthropic Has Agents Vote on Each Other's Code Reviews, Cutting Rollback Rate by 40%

Anthropic has made public the code review setup used by the Claude Code team. Here are three numbers up front: merged PRs more than doubled, rollbacks dropped by more than 40%, and PRs with real review comments rose from 16% to 54%.

The first two numbers are easy to get excited about, but the third is the real point. 16% means that before this system, for more than 80% of PRs, the so-called review was just someone clicking approve. One commenter in the discussion put it bluntly: most reviews are just someone who didn't open the diff giving a thumbs-up.

Here's how the setup works, in five parts.

One: review starts before the PR is even created. Claude runs tests, builds, and linters, and posts the output instead of just replying “done.” Then a second agent reads the diff in a fresh context. The agent writing the code and the agent reviewing it are not in the same context; this point comes up again later.

Two: put a REVIEW.md in the repository. It's written by the code owner and is very specific: what counts as a failure under this directory, at most how many nitpicks to raise, which paths to skip entirely, and that any issue raised must include a file and line number. Move recurring review comments out of the comment section and into this file, so the agent can read them every time.

Three: multiple agents look for bugs in parallel, and another group of agents votes on each finding. Only findings with enough votes get posted, and style nitpicks are discarded outright. One commenter said this is the most surprising part of the whole system: what one agent finds is noise; only when three agents agree is it a signal.

Four: if you're reviewing a PR alone, have an agent with a fresh context go through it again with REVIEW.md, to fill in the parts you skimmed past.

Five: every finding must get a response. Fix it, or write a reason and close it. If there are unresolved findings, the PR cannot be merged.

If you want to copy it, the checklist isn't long:

• Keep CLAUDE.md short, and list only checks the agent can actually run
• Add REVIEW.md and move recurring comments into it
• Run /code-review and /security-review before every PR
• Add three fields to the PR template: Testing, Risk, and How to revert
• Limit review rounds; after a few rounds, only look at newly added diffs

There's a reminder in the discussion worth noting. One commenter said that doubled merged PRs doesn't by itself say anything about quality; it might just mean approvals got faster. What you should really watch is the comment rate going from 16% to 54%. That reminder is right. Another commenter asked about token cost. This process really doesn't save tokens; parallel bug hunting plus voting, the cost is what it is.

Someone pushed the fresh-context idea even further: Claude reviews what Codex wrote, and Codex reviews what Claude wrote. Same-family cross-review is like the same person reading the same code twice.

None of the three things are new: clean context, multi-agent voting, and findings that must be closed out. Combined, the effect is to turn review from a formality into an action with a cost. The agent that writes the PR cannot sign off on its own PR; this may be the easiest rule to copy.

Source: Anthropic blog Solving code review at Anthropic, original tweet Yarchi.