Wink Pings

11,000 PRs, 0.5% Rollback Rate: They First Banned 99% of Comments

Last month, the Grok Bot team merged about 11,000 PRs with a rollback rate of about 0.5%. Their approach is no mystery: turn corrections into lint, ban 99% of comments, and have the AI click Apply itself and submit screen recordings. Build the foundation first; talk about full automation later.

Lauren Tan's talk at the AI Engineer conference, All Code Is Slop (Except Mine), made a counterintuitive point. She is responsible for the desktop version on the Grok Bot team. Last month the team merged about 11,000 PRs with a rollback rate of about 0.5%, and the app is even faster than when it was first released. Someone in the comments checked the charts: October was 10,926 PRs with a 0.47% rollback rate, and June was still 0.98%. The numbers look good, but most of the talk was about how to prevent AI from writing bad code.

Talk video: YouTube

Charts and data from the talk:

image

image

image

image

image

image

Write corrections as lint, not as long documents. Every time you correct the AI, turn that correction into an automated check rule such as lint, and add less to agents.md. Code that violates lint fails directly and cannot be submitted. AI may not follow what the docs say. One commenter said they had piled up several screens of content in agents.md, and the AI still forgot it when it mattered. Switching to lint means it turns red on submit, which works better than reminding it again. Another commenter cautioned that rules become debt. Add one every time you correct something, and after six months, once false positives pile up, people start writing disable comments and loop right back to the comments problem. Rules also need an owner, just like code, and should be deleted regularly.

The codebase is the AI's training set. Bad patterns and temporary patches in it will be copied as conventions by the next AI, spreading like weeds. So the codebase must stay clean. Things that are easy to misuse should simply be banned. There is a React feature that people often use incorrectly and AI is even worse at using, so they disabled it in product code, and that class of problem disappeared. They also banned 99% of code comments. AI loves to use comments to justify shortcuts it took, and later AIs learn from that. When she was at Cursor, these justified patches spread and made the app very slow. One commenter asked whether workaround comments explaining why the usual approach was not used are also banned. That boundary is worth discussing.

Let the AI check its own work. The AI runs the program and clicks through it like a user, then submits screen recordings and screenshots; human code review becomes reviewing those materials. Most of her tokens are spent on this step. Checks that used to be unaffordable are now affordable. Rigorous verification methods such as mathematically proving a program is correct used to cost too much human effort, and AI has brought the cost down, for example TLA+ and Lean. A commenter added that every time a stable version is released, Grok Bot automatically opens a Cursor project, clicks through the entire app as a user, and when it finds a bug, proposes a fix and merges it automatically.

Build the foundation first, then talk about full automation. If the AI code-writing step itself produces bad code, full automation will only produce more bad code and cost more money. Documentation is advice; lint is constraint. When there are so many rules that false positives appear, people write disable comments, and we return to the problem of comments justifying shortcuts. Last month the team merged about 11,000 PRs with about a 0.5% rollback rate, and the app is even faster than when it was first released. The results look good, but the premise is the foundation.

发布时间: 2026-11-16 00:30