Giving Opus a Cheap Writer Dropped Accuracy by 14 Points—and Raised the Bill

Multi-agent architectures have been getting a lot of attention recently. Dividing work, running in parallel, and letting cheaper models write code all sound like ways to save money. Ibragim ran a round of tests on 12 real, difficult tasks, and the results don't quite fit that narrative.

Start with the most striking comparison. Opus 5.5 medium run alone had a task pass rate of 81% at $1.20 per task. Pair Opus with DeepSeek V4.1 Flash 1017 as the code-writing worker, and the pass rate drops to 67% while the cost per task rises to $1.29. The savings never materialized, and accuracy fell by 14 percentage points.

This data comes from Ibragim's HARD12 leaderboard. The test set consists of 12 tasks selected from Terminal-Bench 4, DeepSWE, SWE-rebench, SWE-bench Pro V2, and Longtail-SWE, with task difficulty aligned to his day-to-day work. Each configuration was run 3 times, for a total of 41 configurations and 1,188 agent runs. The tasks and full table are in the leaderboard link.

The backdrop is a long post from DeepSeek Harness on the Super-agent architecture, discussing separating the Router and Worker: the Router manages global context, priorities, dispatch, and replies, while Workers fix bugs, review PRs, and run tests in parallel inside sandboxes, with an independent Watchdog monitoring. The community diagram of the flow is straightforward. The architecture diagram looks like an engineering team with a dispatch center. Ibragim's tests amount to running a real bill for this approach.

The most common main configuration in the experiment had Opus plan and review inside Claude Code, with direct code edits forbidden, while a cheaper model did the writing. Optional workers included Haiku 5.5, GPT-6 Luna, DeepSeek V4.1 Flash, and Qwen3.8 27B. On average, Opus plus a cheap worker solved 55% at $1.25 per task. The best team configuration reached 69%, still not catching Opus alone.

Why teams lose. 71% of team failures occurred on tasks where the worker also failed alone. In most cases, the worker is the ceiling. Some failures also occurred on tasks that both models could solve on their own. One example: the task is to have one agent hand work off to another agent. The specification does not say what to do with empty input, and the hidden tests accept two behaviors: throw a stream error, or run the sub-agent. Opus wrote a ticket for DeepSeek and added a third rule: return an error message but do not throw an exception. DeepSeek followed it. Opus approved, saying all 8 new tests passed. The hidden tests rejected it. Opus alone 3/3, DeepSeek alone 3/3, together 2/3.

Where the money goes. An agent resends the full context at every step, and prompt caching makes repeated reads cheaper. The cache lives for about 5 minutes. Opus waits 10 to 15 minutes for a worker, the cache expires, and Opus rewrites the context. At this gateway's pricing, cache writes cost 25 times cache reads—$5 vs. $0.20 per million tokens. In team configurations, Opus spends $0.52 to $0.84 per task on cache writes, while the entire worker costs only $0.06 to $0.19. When Opus runs alone, cache writes are $0.33. Multi-agent setups turn waiting time into a cache bill.

Some approaches work. Sol reviewing cheap models: DeepSeek went from 56% to 64%, Qwen from 28% to 39%, with review cost around $0.07 per task. A subtler one: let Opus keep writing code but tell it “your tokens are 20 times more expensive than the worker's.” The worker still wrote 94% of the code, and pass@1 stayed at 67%, the same as hard-forbidding Opus from editing code; Opus's own cost fell from $1.19 to $0.59, for a total cost of $0.71. Who the reviewer is also matters. Sol reviewed Opus low, fixed 3 runs and broke none; Opus reviewed Sol, fixed none and broke 4. Self-review helped neither.

The key rows in the table are:

Multi-agent configurations look like modern software engineering; the bill looks like a cloud provider's victory. Ibragim's next step is to try a 1-hour cache TTL for Opus and then measure the total bill. That direction is closer to the actual problem than continuing to pile on workers. After all, in team configurations, the most expensive part is often the one waiting for the work to be written; the cheap writer comes later.

Sources:

• Ibragim's original post: https://x.com/ibragim_bad/status/2117662415965798757
• Full HARD12 table: https://ibragim.dev/leaderboard-v2/
• DeepSeek Harness Super-agent long post: https://x.com/i/article/2116829218935857152