Average scores went up, but the Agent got worse

Day-to-day work tuning an Agent usually starts with manually trying a few questions. You change one version of a prompt, ask a few examples, feel that it is better than the previous version, and ship it. A couple of days later, a user reports a strange case; you dig through the logs and discover that the last change broke another feature. The problem is not the prompt itself; it is the lack of fixed measurement.

Eval-driven development (EDD) borrows its name from TDD. TDD first writes a failing test, then writes code to make it pass. EDD first defines the behavioral standard for the Agent, then writes evals, and finally climbs toward the evals.

Agents are different from ordinary code: outputs vary, and the same question may get a different answer each time; the same question may have multiple correct answers; and the Agent decides which tools to call and which steps to take. Traditional assertions are not enough. Evals score a set of examples and can check the final answer as well as tool calls, intermediate steps, and system state. Deterministic checks use code, and fuzzy judgments use LLM-as-a-judge. LangChain blog author Divya Santhanam breaks this workflow into three steps: define what good means, write evals first, and make changes and compare them.

Step 1: Define what good means

Do not write criteria that cannot be scored, such as 'the answer is helpful.' Choose one behavior and write a short spec: when it should trigger, what it should do, what it should not do, and which existing behaviors must be preserved. The spec must be able to judge pass/fail.

Example: an internal docs Agent has search_docs and web_search, and now you want to add create_ticket. The spec is: when a user reports that something is broken, such as a laptop cannot connect to the VPN, call create_ticket; for how-to questions like 'How do I connect to the VPN?', do not create a ticket; search_docs and web_search should continue to handle what they handled before.

Step 2: Write the eval first, then the feature

The dataset should come from real user requests; do not make it up. Real traffic contains phrasings and edge cases you did not anticipate. LangSmith can add online runs directly to a dataset. Each example has an input and may also have a reference output.

Then write the evaluator: use code to check deterministic things such as tool selection; use an LLM to score open-ended quality. Run the current Agent once and confirm that the new eval fails on the old version.

Step 3: Compare after making changes

First run a baseline and record the current scores. Then, after every change, run a separate experiment and compare it example by example with the previous one. This is called hill climbing: take small steps, and have evidence for every step.

LangChain's example is typical: after adding create_ticket, the ticket-related examples pass, and the average score goes up, but 'How do I connect to the VPN?' changes from looking up docs to creating a ticket. The average score masks the regression.

Open the trace to see why it failed, and you find that the tool description for create_ticket is too broad and includes 'VPN and laptop issues.' Narrow the description, run it again, and everything passes.

Here is a counterintuitive point: do not chase 100%. If you tune on the same set of examples for too long, the Agent will overfit to the dataset. Keep a held-out set of examples and do not look at it during iteration. If your tuning set improves but the held-out set does not, that is overfitting.

If the Agent changes the environment, such as writing code in a sandbox, checking only tool calls and the final answer is not enough; you need to check the system state it leaves behind, and use a clean environment for every run. LangChain uses Harbor to run these kinds of tasks, and Terminal-Bench uses it too.

Zal Zhou mentioned a detail in a reply: EDD is useful not only when changing prompts; you should also run evals when changing tool definitions or context rules. Many regressions come from tool definitions or context rules, and a suite that only tests prompts will not see them. This observation is worth remembering.

Do not stop after launch

Put fast checks in CI and run them on every PR; run slow LLM judges every night or before a release; add online evaluators in production to score samples; use a human labeling queue to check judge-human agreement; and feed production failures back into new dataset examples. LangChain has open-sourced the eval suite for Deep Agents, so you can see directly what real production evals look like.

Here are common mistakes:

• Success criteria are too loose. 'Helpful' is not a metric.
• Write the eval after seeing the fix. That makes it easy to design the eval around the result, so it is bound to pass. Write it first and confirm that it fails on the bad version.
• Look only at the average score. The average may go up while an individual case regresses.
• Add a pile of unrelated evals. Every eval should be tied to a specific change or failure.
• Trust an LLM judge that has not been validated. First compare it against human ratings, ask narrow questions, and use pass/fail rather than a vague 1-to-10 score.

EDD is not new; TDD has been around for a long time. What is new is that the nondeterminism of Agents makes 'it feels better' especially unreliable. Fixed evals, example-by-example comparison, and keeping regression cases decide more than switching to a more expensive model whether an Agent can be iterated on over the long term.

Original article: What Is Eval-Driven Development? How to Improve AI Agents

Ajitesh Shukla's thread: link

Deep Agents eval suite: GitHub