That Bug That Passed All Tests Is More Worth Reading Than AI Writing Code Itself

Anthropic recently added a tool called advisor to the Claude Code documentation, allowing you to have the main model consult a stronger model at key decision points. Someone packaged it into a complete cost-optimization scheme: Sonnet 5.5 runs the main session and writes code, Haiku 5.5 explores files and documentation in parallel, and Opus 5.5 acts only as an advisor, stepping in at three moments—before locking in the plan, after the same error appears twice, and before declaring the task complete. The command is just one line: claude --model sonnet --advisor opus.

The logic is straightforward: don't use Opus for work that doesn't need Opus. Someone in the replies did the math: with Haiku doing context mapping, Sonnet's context window can stay clean and won't get stuffed with noise when pulling references. Others poured cold water on it: what decides the model switching? Manual or routing? Opus 5.5 still makes mistakes, and the errors from Sonnet and Haiku will only quietly pile up.

None of these discussions, however, touched the real crux.

On October 14, NO1ennn published a long article about REA, a reverse-engineering tool. REA is an MCP server plus CLI that lets coding agents understand programs without source code, all the way down to the machine-code level. The part of the article most worth reading is not the installation steps, nor the slogan 'software is dead,' but a test that had already passed in the TH08 project.

A function had already passed a byte-for-byte comparison. Then a downstream console port reported an item auto-collection bug. The reconstructed code compared player firepower against 0.0, while the original used 128.0, the maximum firepower threshold. One constant, a different game.

Why did it pass? The comparison tool adjusts addresses before comparing instructions, because the reconstruction may place the constant elsewhere. But it did not check the floating-point value stored at that address. The instruction bytes matched; the meaning did not.

The fix had two halves: change the source code so the comparison tool checks the actual bytes of constants. Then audit everything that passed weak checks—1,548 floating-point literal references turned up another twelve errors, spread across five functions that had already been 'accepted.'

NO1ennn quotes the original author N0zoM1z0: 'Test the oracle too.' A verifier is software too, and it has bugs. A verifier with a blind spot won't let just one function through; it will quietly approve hundreds.

This and the logic of the advisor tool are actually two sides of the same thing. Splitting tasks among different models is essentially admitting that no single model is reliable at every stage. But if the verification step itself has never been verified, then even with Sonnet writing, Haiku checking, and Opus reviewing, the reviewed result can still be wrong.

In REA's own cases, an audio left-right balance function in DX-Ball passed all 3,205 test cases, and 63 compiled bytes were reproduced exactly. That is what a behavioral verifier can do. For the Windows Calculator percentage button, the rule that 200+10%=220 was cross-validated against the corresponding branch in Microsoft's public source code. That is treating the explanation itself as an object of verification.

Conversely, that floating-point story shows: what you can trust is determined by the verifier's coverage. Byte comparison covered instructions, not data. Passing your tests does not mean your tests actually covered the thing.

So if you want to use the Sonnet+Haiku+Opus combo in a team, it's worth doing one thing first: add three deliberately failing cases to the verifier itself. Change a constant, swap a branch, move a boundary. If all three faulty variants pass your checks, then none of the green results that follow can be trusted.

The Advisor tool solves the cost of decisions. The Oracle problem solves the cost of truth. The former saves you money on Opus; the latter saves you the time of rewriting the entire project. Only by doing both do you close the loop.

Anthropic's documentation is very restrained: advisor is an experimental feature, and its behavior, pricing, and availability may change. NO1ennn's article ends with this too: what Berman said about 'software is dead' is only half right. What ended is the assumption that 'closed source equals unknowable.' What has not ended is the difference between reading and taking.

And what determines whether all this holds up has never been the model. It's the verifier—and whether you have tested the verifier.