Agents Are Stable in Demos—Until Users Start Using Them
Plurai Simulation launches, free for AI engineers. It clones an agent's tools and environment into a digital twin, stresses edge cases with simulated users and tasks, and can turn production failures into replays. Replay is practical, but simulation fidelity and the grader determine whether it's useful.
Plurai Simulation is live, free for every AI engineer. Ilan Kadar said it plainly in a tweet: AI agents look great in demos until real users break them. Plurai's approach is to build the agent its own world, so it can fail, learn, and get fixed before users see it.
Features include:
- Digital twin: rebuild the real agent, tools, and environment from code.
- Simulated users and tasks: push the agent into edge cases using realistic workflows.
- Automated evaluation: check what the agent actually did, not just what it said.
- Verified fixes: each failure comes with the run that exposed it and a verifiable fix.
- Replay: turn production failures into simulations and replay them against new versions.
It runs inside existing coding agents: Claude Code, OpenAI Codex CLI, GitHub Copilot. The free entry point is at simulation.plurai.ai/connect.
The part worth watching is replay. Turning production failures into replayable tests is closer to how teams fix bugs than piling up scenarios in a demo. Commenter Farhan said replay is his favorite part: turning real production failures into tests and rerunning them against new versions is exactly what most agent teams lack.
But there are two pitfalls. Commenter Alano warned that the simulated environment's state and permissions need to be close to production; otherwise the agent may repeatedly pass the same set of scenarios while the real integration has already changed underneath. Commenter Andrew Rose also posted a scoring error on MATH-500: it ran 76.0%, while the actual was 79.2%, and 16 so-called misses were just formatting issues. No matter how complete the digital twin is, a wrong grader still gives false confidence.
In the comments, others also asked how repo scan decides which endpoints are mocked and which stay live, especially when a single internal tool handles both RAG reads and writes. That question is not trivial; simulation fidelity can't be waved away with a mention of a digital twin.
The value of Plurai Simulation is letting agents step in all the potholes in their own world before going live. It's free to start, and at least worth a try.
发布时间: 2026-11-12 14:28