AI Coding Agents: Fast Doesn't Mean Less Work
EveryDev.ai: AI coding agent benchmarks miss the most important thing. Test Claude Code and Codex on the same bug, three runs each - the fastest one isn't necessarily the winner.
EveryDev.ai took issue with something on X: AI coding agent benchmarks all compete over who runs faster and who passes the tests, but no one cares whether you'll have to redo the work after the fix.
Claude Code and Codex are currently representative AI coding agents, from Anthropic and OpenAI, respectively. EveryDev.ai suggests testing them this way: pick the same bug and have each agent fix it. For every run, record four numbers: time to pass tests, token cost, how many manual edits are needed, and whether the result would ultimately be merged into the codebase. Run each agent three times.
Of the four metrics, time is the easiest to be misled by. Even if an agent finishes the tests in seconds, would you dare merge it if the generated code requires half an hour of manual correction, or if token consumption is absurdly high? Conversely, the slower one may get it right on the first try, and you can sign off after a quick check.
Why run three times? Because models are stochastic. A single result could be luck; only after three runs can you see the stable level. Also, each agent has its own personality: some are good at producing quickly but always fall a little short on details; others quietly put in a few extra steps but produce cleaner results.
EveryDev.ai's conclusion: the fastest agent is not always the winner. Benchmarks are not the same as real-world development experience. Next time you look at an agent evaluation, don't just stare at speed numbers; ask more about the amount of manual correction and token cost. Those two are the real bill.
This tweet didn't include specific data, but the method itself is a reminder: as competition over tools gets more and more intense, evaluation criteria should return to the most fundamental question: would you be willing to ship this result?
发布时间: 2026-11-20 23:33