The Same 27B Model Scores 3.8× Higher on OSWorld After a Change of Interface

Simular posted a demo on X: it connected Codex to Sai MCP, gave it a plain English task, and Sai opened Calculator in a hosted Windows desktop, entered the expression, and left two results in History. The entire action takes place in a real application, not through an API call.

For developers, the interesting part of this demo isn't the calculator. It hides desktop control behind MCP. MCP is the Model Context Protocol, which exposes tools to agents through a standard interface layer. Previously you had to build your own VM, screenshot pipeline, action loop, and retry logic; now you can hand that job to a hosted desktop. Coding agents gain a pair of hands that can click a mouse.

But a demo is just a demo. Simular itself notes that this video does not reproduce the Agent S4 experiment, and the waiting time was cut out. That note is necessary, because the next set of numbers comes from a different evaluation.

Agent S4 studies interfaces. It defines what the agent sees and what operations it can perform as an interface: the GUI interface receives clicks and keyboard input and returns screenshots; the shell interface receives commands and returns text. Change the interface, and the task difficulty changes. Making an agent edit a spreadsheet cell by cell using screenshots is not the same as giving it an interface that writes cells in batches.

Simular's argument is that computer-use agents are stuck with fixed interfaces designed by humans. With the model and harness held fixed, changing only the interface repository can still raise the score. Agent S4 uses population-based evolution to have a meta-agent read task-agent trajectories, write new interfaces and usage knowledge on synthetic training tasks without success labels, and then use validation tasks unseen by the meta-agent to decide which repositories continue to reproduce.

On OSWorld V2, Qwen3.8-27B with the evolved interface repository reaches 21.3% binary success; the same model with a fixed interface gets 5.56%, a 3.8× increase. Cost is $1.10 per task. GPT-5.5 is 13.0% at $25.50 per task; Claude Opus 4.8 is 20.6% at $72.40 per task. These are vendor-published benchmark numbers and should be considered separately from the calculator demo at the beginning.

Some commenters in the replies focused only on the gap between 1.10 and 72.40. That gap is indeed striking, but what's more interesting is where the score increase comes from. The model didn't change, the harness didn't change; what changed was the set of operations the agent could call. Some interfaces appeared in the evolved repository that no one specified, such as writing an entire block of LibreOffice cells at once, sending requests in a logged-in browser tab and reusing it across email, banking, and chat sites, and adding a command interface to ChimeraX.

The run with a scratch VM had a higher partial score, 51.06%, while binary success was instead 19.44%. Being able to try code helps, but the best binary result came from a different run. This detail is more worth looking at than a single overall score.

The same method also works on ScienceBoard. Its 169 tasks are spread across six scientific applications: Celestia, ChimeraX, GRASS GIS, KAlgebra, Lean, and TeXstudio. The meta-agent learns from 48 synthetic tasks, and the evolved repository raises the same model from 41.4% to 58.6%, solving 99 tasks. General-purpose computer-use interfaces rarely cover these applications, so the room for interface evolution is even more obvious.

This kind of thing is also useful for vertical agents. If repeated actions can be compressed into one operation, both tokens and latency will drop. Simular's platform page offers another productization angle: the Sai API provides Windows or Linux cloud computers, MCP access, repeated tasks stored as code for replay, and when the UI changes, an independent judge checks the screen and lets the agent take over; it claims repeated runs can save up to 90% of tokens. Vendor claims need to be verified yourself, but the direction is clear: put the dirty work of VMs, screenshots, action loops, and approvals into the infrastructure.

Simular also reports that Sai with Claude Opus 5 reaches a 79.38% partial score on OSWorld 2.1 at $14.34 per task. These are likewise vendor numbers and should be considered separately from the calculator demo.

Back to the beginning. Giving Codex a Windows computer has its most direct value in letting coding agents handle old software and login pages that have no API. The calculator demo is small, even a little funny. But the path it shows is not small: an agent does not need an API before it can get work done; the desktop itself can become the interface. Agent S4 goes further and says interfaces can also be evolved. Model leaderboards will keep changing, but this axis of interfaces may be underestimated.

Paper: https://arxiv.org/abs/2610.28601
Code: https://github.com/simular-ai/agent-s
Sai API: https://platform.simular.ai/