The AI That Grades Itself Always Passes
Anthropic Labs gave Claude one sentence and got back a complete app in 6 hours. The model didn't change; the process did: three agents, and no one grades themselves. An agent that grades itself always passes; an unfamiliar reviewer using Playwright does not. The same logic applies to your AI bill.
Anthropic Labs published a harness paper packed with information.
They gave Claude a single product requirement and got back a complete app 6 hours later. The model didn't change—Claude is still Claude. What changed was the process: three agents divided the work, and not one of them was responsible for grading itself.
prompt → spec → contract → build → QA → fix
Six stages with clear boundaries:
1. Planner: Reads 1–4 sentences of input and produces a spec with 16 feature points. It stays at the product level and contains no implementation details.
2. Sprint contract: Before writing code, the generator and evaluator go back and forth in a file to define 'done.' Work only starts after this step is complete.
3. Generator: Each sprint does one thing. After a self-check, it hands off to QA.
4. Evaluator: Uses Playwright to click through and verify the app while it is actually running. The third sprint has 27 acceptance criteria.
5. Hard thresholds: Depth, functionality, design, and code. If any one fails, the sprint is marked as failed.
6. Cost: Solo: 20 minutes, $9; the game does not respond to input. Harness: 6 hours, $200; the game is playable.
An agent that grades itself always passes. An unfamiliar reviewer using Playwright does not.
The reason is not complicated. Self-evaluation means the standard can be quietly lowered after failure. External evaluation means the standard is written into the contract before work begins and cannot be changed midway.
Someone in the comments asked: Who checks the QA agent? That question gets to the root. There must be a human standing at the end of the evaluation chain.
This logic does not only hold true for the agents in the paper. In another long post, rari (@0xwhrrari) brought it into everyday AI workflows and called it Mode Engineering. The core observation: your AI bill is not a question of model prices; it is a question of how much work you send to top-tier models.
Mode engineering routes work by type:
Judgment work, read by humans → good models
Strategy questions, replying to clients, writing launch copy. Quality is visible, and mistakes are public.
Routine work, read by humans → cheap models
Daily summaries, standup recaps. The shape is the same every day, and no top-tier model is needed.
Text no one reads → cheapest model
Tagging, classification, field extraction. You are paying for sentences no one looks at.
Has a single correct answer → code
Sums, queries, dates, thresholds. This is not work a model should be doing.
Key move: set modes on channels, not by choosing each time you have a conversation. Work is already divided by channel—the research channel is for hard problems, the standup channel is for summaries, and the inbox channel is for tagging. With one default mode per channel, choosing a model goes from a daily decision to a quarterly one.
Two more levers:
Nothing to do → don't run
The most expensive run is an idle one. Check once an hour, find nothing 23 times, but pay a top-tier model bill every hour. A conditional script can skip the idling: the model only starts when something new appears. The original post gave an example: 720 AI runs became 72. The 648 runs saved were script checks and cost nothing.
Fixed steps → script them
The daily morning data summary should not make a model read 40 lines of data and do mental math. The script pulls the data, sums it up, compares it, and hands five numbers to the model; the model only writes the conclusion. Shorter runs, lower cost, zero errors.
Your AI bill is not a question of model prices; it is a question of how much work you send to top-tier models.
First ask: does this step need a model?
发布时间: 2026-11-21 03:42