After the Agent Added a New Tool, the Average Score Rose—but VPN Inquiries Started Creating Tickets Incorrectly

LangChain recently published an article about Eval-Driven Development, and Xudong Han recommended it in a tweet. The article is about integrating Eval into the day-to-day development of Agents, using a stable set of metrics to determine whether each change is actually useful. The original author is Divya Santhanam, with the link at the end of this piece.

Testing an Agent is different from testing ordinary code. The same question may get a different answer each time, two different phrasings may both count as correct, and the Agent also chooses its own steps. It may call one extra tool, or miss calling a tool, and the final answer may still look fine. EDD borrows the idea of TDD: write the evaluation first, then optimize toward the evaluation. The article breaks the process into three steps: define what counts as success, write the Eval before making changes, and after the changes compare against the same set of tasks.

The first step is the hardest. You first need to write down clearly what the Agent should do, what it should not do, and which existing behaviors must be preserved. The article uses an internal documentation Agent as an example. It originally had search_docs and web_search, and the plan was to add create_ticket to handle IT tickets. The spec is written very concretely: when a user reports broken equipment, such as a laptop that cannot connect to VPN, it should call create_ticket. When a user asks how to connect to VPN, that is a how-to question, and no ticket should be created. Questions handled by the existing search tools must continue to work normally.

Step two is to first build a dataset and evaluators. The inputs in the dataset should ideally come from real runs, because users will come up with phrasings and edge cases you did not think of. The article pairs each example with a reference output, that is, the tool the Agent should call. Tool selection can be checked with code, so a code evaluator is used here, without rushing to bring in an LLM judge.

First run the Baseline. The new tool has not been implemented yet, so ticket-related examples will fail, while the other examples record existing capabilities. After making changes, run the same set of Evals again and compare the experiments.

Step three: the example in the article deserves a separate discussion. After adding create_ticket, the overall Eval score improves, and ticket requests pass. Then the problem appears: when a user asks “How do I connect to the VPN?”, the Agent starts creating a failure ticket. The average score went up, but a scenario that was originally working got broken.

Only by opening the Trace can you see the reason clearly. The tool description for create_ticket said “VPN and laptop issues”, which is too broad. As long as a question mentions VPN, it may hit this tool. After narrowing the description to requests that require actual IT handling, run the Eval again, and all examples pass.

This part is easiest to misread. 100% passing on a small dataset does not mean you can wrap up. The tool description was tuned by looking at this set of examples, and continuing to tweak it may lead to overfitting. The article suggests keeping a held-out set that you do not look at during iteration. If the tuning set score rises while the held-out set stays flat, that suggests possible overfitting.

Some commenters added several practical points. @ai2face said that in addition to looking at traces, it is best to give new tools a batch of negative test cases that are “strictly forbidden to call”, specifically to prevent collateral damage. @老王实验室 suggested making “ticket miscreation rate for ordinary inquiries” a separate release gate, rather than averaging it together with “successful ticket creation rate for failures”. If the total score rises but the miscreation rate increases, do not ramp up yet. The proportion of the two types of requests in the evaluation set should also be recorded; switch to a different sample and the conclusion may be completely different. @Lucky 沐晴 also mentioned that the false positive rate must be gated separately, and not fooled by the average score.

@oyung's judgment is more engineering-oriented: TDD has strict inputs and outputs, while EDD also has to compare the process; every tool call and context injection has to be examined, and designing the verification environment is actually harder. @Martin said that defining what counts as success is the hardest step, and many Agent projects get stuck because they cannot establish acceptance criteria. This matches the beginning of the article. In the early days of an Agent, manual testing can still help you judge; after many changes, without stable Evals, it is hard to know whether the latest change is an improvement or a regression.

The article also mentions that Evals should run in the release process. Fast code checks can block bad merges, while slower LLM-judge evaluations can run overnight or before a release. After launch, you can use online evaluators to score samples, use a labeling queue for human review, use a Dashboard to watch scores, cost, and latency, and use Insights to cluster production Traces into new issues and then turn them into dataset examples.

Common mistakes are also listed directly. The definition of success is too loose, for example “the answer is helpful”, which cannot be judged as pass or fail. Writing Evals after fixing a problem makes it easy to write tests that are guaranteed to pass. Looking only at the average score misses regressions. Adding a bunch of unnecessary Evals means each evaluation pushes the Agent in some direction. Using an LLM judge without aligning it with human scores is also quite risky.

EDD is worth putting into the regular Agent development process. Adding a new Skill, changing a Prompt, adjusting a Workflow, or deleting an old Harness should all use the same set of Evals to determine whether the change is truly effective. The original article is on LangChain Blog, and Xudong Han's tweet is here. A good-looking average score is only the surface; the regressions in the details are what you need to watch.