Merging 30,000 Code Changes a Day: Anthropic's People Are Starting to Watch Screen Recordings

In May of this year, Anthropic was merging about 2,500 code changes into its codebase each day; by September, that number had exceeded 30,000. The figure comes from an 18-minute conversation with Addy Osmani at the AI Engineer conference. Addy worked at Google for about 14 years, where he long led Chrome's developer experience, and joined Anthropic in September of this year to lead developer experience for Claude Code. The interviewer, Manish Kapur, leads product and developer relations at Sonar, which makes code quality and security inspection tools.

Addy also mentioned the redesign of the Claude.ai web app, whose performance roughly tripled, with about 3,000 PRs merged in a short period. The people involved were not only engineers but also product designers and people in other roles. With 30,000 merges a day, it is impossible for humans to read code line by line. Addy described a review process in several steps.

First, Have Claude Use It Itself

The first step is to have Claude review it first. The key is to make it actually use the feature like a real user: run end-to-end tests, open pages in a browser and inspect them, and run various tools and lint checks. When it finds bugs or feature gaps, use Hooks—scripts that trigger automatically at specific moments—to run checks automatically every time the code changes. Anthropic's own code review product is also brought in. The overall idea is to automate code review as much as possible.

The second step is to write down the team's taste. Addy says teams should write the best practices they believe in into shared files such as CLAUDE.md and AGENTS.md. These files live in the project and are written specifically for AI to read; the AI refers to them when reviewing code.

Only the third step involves humans. People are responsible for checking the evidence the AI provides. If you're building an interface users use in a browser, have the AI test it in the browser, ideally generating a screen recording of it clicking around. If you care about the mobile or desktop experience, have it provide screenshots or recordings on those devices. Addy's phrasing is that the goal is for the AI to provide as much verification evidence as possible, so that when people have evidence, they dare to say it works.

The most counterintuitive part here is that the object of review has changed. In the past, review meant looking at the diff, reading code changes line by line. Now, in Anthropic's process, people look at the chain of evidence the AI submits: test results, browser recordings, device screenshots, and run logs. Thirty thousand merges sounds like an assembly line, but when it comes to review, it's more like accepting a chain of evidence. What is saved is the time spent reading code; not a single verification step is skipped.

The Bottleneck Has Moved

Why has verification suddenly become important? Manish offered several sets of numbers. The continuous integration service CircleCI analyzed 28 million automated pipelines this year; code output rose 59%, but 3 out of every 10 attempts to merge into production failed. In an upcoming Sonar survey, 76% of developers who use AI to generate code said code output is so fast that they simply don't have time to verify it. The writing-code step is getting cheaper; the verification step is not.

Addy says the thing technical leads care about most now is how to ship faster while maintaining confidence. In the past, saying a piece of code was high quality was based on good tests, performance benchmarks, and requirement specs. These things that give people confidence can now be generated cheaply and quickly by AI, and then people revise and refine them. He believes the place engineers should spend the most time now is checking. The next improvement won't come from models writing code, but from verification. Addy believes model quality is already quite good—he cites Opus 5.5 as an example—and that there is more room in verification.

Verification needs to be concrete. If you tell someone to 'verify it,' their next question is what verification actually means. Anthropic uses three main types most often: end-to-end tests, which simulate a user going through a flow from start to finish; property-based tests, where instead of writing individual concrete examples, you write a rule and let the program automatically generate large amounts of random input to verify it; and visual verification tests, which use screenshot comparison to check whether the interface has changed. The goal is to build enough confidence for these changes to merge automatically.

From Sonar's perspective, Manish added a multi-layer verification framework advocating zero trust. The first layer is automated scanning with fixed rules, checking for security vulnerabilities such as SQL injection and common bugs. The second layer is AI review, checking for logic issues that rule-based scans cannot catch and whether the code matches its original intent. The third layer is software supply chain checks, looking for known vulnerabilities or malicious code in the third-party packages the project depends on. Each layer catches a different class of problem.

Software Factories Come in Lights-On and Lights-Out

The conversation's title includes the term 'software factory,' which comes from an article Addy wrote this year. His definition is scaling the loop: having AI plan, write, check, and ship code over and over again. Software factories come in two types. In a lights-on factory, AI handles these steps, but humans are still in the loop, using their judgment and taste to gatekeep. A lights-out factory is more like a fully automated car factory: robots keep working, humans retreat further upstream and are only responsible for setting direction, and are called in only for the most exceptional cases.

Addy says most teams are somewhere between the two, figuring out how to increase automation step by step while maintaining trust. The more verification AI can provide and the more confidence it creates, the further they can move toward lights-out. Sonar's survey shows that currently only 5% of companies are using software factories, with that expected to rise to 34%; in financial services, it's 39%. Asked what kind of factory most teams would be by the end of 2027, Addy's judgment is mainly lights-on factories, with some lights-out corners.

As automation increases, governance becomes an enterprise challenge. Addy says many companies he spoke with a few months ago were worried about the same thing: they let engineering teams use AI agents, and then hundreds or even thousands of agents appeared, with no one knowing what they were doing. They access various systems and data and may even be deploying to production. Enterprises care most about three things: how to control them, how to allocate permissions, and how to use sandboxes to ensure agents don't cross boundaries.

Should AI-written code be labeled in PRs? Addy thinks so. In the past, engineers had a strong sense of ownership over the code they wrote; at Anthropic, people care less about Claude being credited as the author and more about whether what is delivered to users is right. Labeling has another use: knowing it was written by AI lets you decide what additional constraints and checks to add. Manish agrees; at least for the next six to nine months, it should be labeled, and for financial clients it also matters for auditing. Both emphasize that accountability does not change. Addy says that even if Claude did the work this time, if something goes wrong, he is still responsible and still has to fix it.

A commenter added context: Anthropic replaced Git with a self-built version control system called Scion, and its code review tool is called Antique; seven people built it in five months, and the entire monorepo was migrated over. This reply is a reminder of one thing: if other companies want to copy the idea of people only looking at evidence, they first need this foundation.

A developer in the comments had tried a similar approach: write a few end-to-end cases with Playwright in Cursor, run them with the screen plugin to automatically generate recordings and screenshots, and paste them directly into the PR. UI changes can basically pass review this way, but pure-logic PRs still need property-based tests to inspire confidence. Another commenter put it more bluntly: evidence has to be cheap, cheap enough that you're willing to look at it every time. If evidence is expensive, people will eventually skip it.

If You Want to Adopt It, Don't Turn Off Review Yet

So if you want to read less code, don't rush to turn Review off. First add the step Addy described: have AI attach evidence when it hands in work. If you don't know where to start, start with the interface most likely to have problems, and require it to actually operate it in a browser and give you screenshots or recordings. You still have to look at the evidence yourself; it's just that watching a recording is much faster than reading hundreds of lines of code.

Manish's three-layer verification framework is worth referencing, though Sonar itself sells code inspection tools, so keep that in mind when listening to that part.

In the rapid-fire Q&A, Addy's advice to technical leads is to push their teams to think clearly about what verification specifically means, where testing currently stands, and what is still missing before AI can judge on its own that code can merge and you feel comfortable with it. For developers just entering the field, accumulate more experience with end-to-end tests and property-based tests.

Video source: https://www.youtube.com/watch?v=AnsXWH2_EBU

Original tweet: https://x.com/dotey/status/2118164405594595702