Turn Agent Failures Into the Next Round of Evals

For an Agent to get better, it first has to see where it is failing.

In a tweet, Kartikeya Gupta breaks the path into two steps: first use traces to automatically grow evals, then use those evals to tune the harness. NeoSigma's NeoEval focuses on the first step. It starts from production data: real tasks, failure records, and human corrections. According to product lead Gauri Gupta, this data contains what it takes to judge a successful run, and over time it becomes the team's own institutional knowledge.

What NeoEval does is turn this experience into eval environments, with environments that stay close to real tasks, workflows, and edge cases. The Agent can replay in its own environment, and the team then checks whether it passes, evaluating and optimizing against the metrics the business actually cares about. The process sounds plain, but the value is in the closed loop: production failures are captured into evals, and evals then become the target for tuning the harness. Over time, production experience starts to compound, and the team ends up with its own intelligence lab.

Of course, the product video and tweet present a vision; the results will depend on integration cost, trace quality, eval stability, and whether the replay environment can stay close to production. For teams building agents, the direction is clear: first turn real failures into evals that can be run repeatedly, then talk about tuning prompts and harnesses.