Read the trace first, then write the eval: what a $320 hotel reimbursement exposed about evaluation blind spots
Many Agent evals have scores and dashboards, yet users still get wrong answers. Hamza Tahir uses an expense review Agent to demonstrate a clumsier but more reliable process: first read traces to find failure modes, then write evals, and finally test the evals themselves.
Many teams already have evals. The scores look fine, and the dashboards look fine. Users still get wrong answers.
The problem often lies in where the evals come from: checks are written first, and only then do people look at how the Agent actually fails. The checks test problems that are easy to imagine and miss the places where it really goes wrong. Fewer teams go on to ask one more question: is the eval itself correct?
Hamza Tahir breaks down this process in a 3,500-word X article. He is a co-founder of ZenML Labs and is building Kitaru, which turns Agent traces into reliable evals. Original link: https://x.com/i/article/2123832962718355457
He built an expense review Agent with LangGraph, using gpt-5-nano, five tools, and review of reimbursements against ten sections of company policy, outputting approve, reject, or escalate to a human. Then he read 126 traces, wrote evals for the failures he saw, and tested the evals. Along the way he found that two of his own evaluators had problems.
One expense report passes every conventional check, and the conclusion is still wrong
The case is very concrete. An employee submits a hotel bill for one night at $320 in Port Selene. Company policy sets a per-night cap of $250 for standard cities and a per-night cap of $350 for Port Selene. The Agent escalates the reimbursement to a human because the amount check returned a $250 cap.
That night did not exceed the Port Selene cap. The flag was wrong.
But all the conventional checks pass: was the tool call correct? Yes, it looked up the expense record and searched the policy. Did it retrieve the correct policy? Yes, the lodging clause. Did it cite a real clause ID? Yes. Was the reply to the employee reasonable? Yes, it had a decision and a reason.
Every check passes, and a compliant $320 bill still goes into the human queue. Looking only at the final reply, it is hard to know what broke. You have to read the trace.
A trace is the step-by-step record of an Agent handling one request: what the user said, the inputs and outputs of each model call and tool call, what the tools returned, and what the final reply was. The original article places GIF demos in sections such as capturing traces, code evaluators, and local runs. The source material did not provide embeddable GIF links, so the text description is retained here.
Eval is not a synonym for LLM scoring
Hamza defines eval narrowly: a repeatable check on a behavior you care about, with evidence for judging whether it passed.
Repeatable means that after changing the prompt, tools, or model, you can run it again on the same set of cases. A behavior means you can state clearly what is being checked. Evidence means someone can open the record and verify the conclusion. He specifically warns against conflating evals with LLM-as-a-judge.
The process has seven steps. The first two are done once; after that, every time you change the Agent you have to rerun them, because every run produces new traces.
- Define the work: who depends on the Agent, and what result each party needs.
- Design cases: define dimensions first, then write synthetic requests.
- Capture and read traces: read from the request all the way to the result.
- Identify failure modes: note each problem one by one, then group them.
- Choose an evaluator for each failure mode: use code when evidence can be checked; use an LLM judge when judgment is needed.
- Test the evaluator: confirm that it fails known-wrong cases and passes known-correct ones.
- Change the Agent and rerun: see what got fixed and what got broken.
First define who is waiting for the result
The expense review Agent serves three groups. Employees need a decision and an accurate reason. Finance needs a decision record that can be stored and reconciled. Human reviewers need to be told clearly which reimbursement needs review and why.
All these obligations need to leave evidence in the logs. The Agent's instructions require it to call recommend_decision once per review, and if the decision is to escalate to a human, to call flag_for_review as well. It can still miss a call or make it twice.
Across 126 runs, there were 23 cases where the employee received a decision but finance had no matching record. In 21 cases the decision was not stored at all, and in 2 cases the stored decision was inconsistent with the reply. The employee had an answer; finance had nothing to act on.
Do not generate 150 cases all at once
If the Agent is not yet live and there are no real user traces, start with synthetic requests. Hamza has seen a common approach: have a coding agent generate 100 to 150 queries in one shot. On closer inspection, many are just rewordings covering the same scenario, while edge cases are missing.
A more reliable approach is to define dimensions first. Dimensions are details you vary deliberately, and they may come from the applicant, the policy, or tool capabilities. On the first pass, have the coding agent propose dimensions and example values based on the instructions, policy, and tool descriptions. Humans review: are simple cases covered? What happens when a query returns an empty result? Are both approval by the wrong person and missing approval included?
Once the dimensions are right, combine them into scenarios. For example, employee, hotel stay, missing invoice can become a single request: 'I lost my hotel invoice. Can I still get reimbursed?' On the second pass, have the coding agent write requests for the selected combinations. Finally, review manually for duplicates, gaps, and wording that does not sound like something a user would say.
This small Agent did not need 150 cases. Hamza used 42 requests and ran each one 3 times. The model does not answer identically every time, and a single run can hide failures. 42 requests times 3 runs gave 126 traces, and he read all of them.
Production is another order of magnitude. For an Agent handling 1,000 to 10,000 requests per day, a practical starting point is to randomly sample 150 to 200 traces from the past seven days and read them closely. When looking for rare failures, sample for them separately and deliberately; random samples will miss them.
Reading a trace means checking whether the action actually happened
When reading a trace, first look at what the user asked and what they were finally told. Then follow the calls in order and check what information the Agent had at each step and whether the actions it claimed to complete actually happened.
In the Port Selene case, the query showed one night at $320, the policy search returned the lodging clause and the $350 Port Selene cap, but the amount check returned the standard-city $250 cap. Putting the three steps together shows that the Agent had the correct record and the correct policy, but trusted the wrong cap returned by the tool and flagged compliant lodging for escalation.
While reading, leave short notes at the point where something goes wrong. State what the Agent did and why the evidence shows it was wrong. Do not just attach a label like 'wrong answer.' 'Wrong answer' is too coarse; it cannot tell you whether to fix retrieval, factual understanding, or rule application.
Group similar notes and name the patterns. Among Hamza's 126 traces, there were 9 wrong decisions spread across 7 requests. Grouped by the main problem: 5 sent clear violations to human review, 2 used the wrong limit or unit, and 2 approved reimbursements that should have been blocked.
Another 23 were not decision errors but missing or misaligned finance records. They count as a separate failure mode.
If you can check it with code, do not reach for a judge first
After listing the failure modes, ask one by one: what evidence in the trace would show that this happened?
If the evidence can be checked directly, use a code evaluator. For example, recommend_decision returns a confirmation containing the reimbursement ID and the decision content. A few lines of code can check whether it was called exactly once and whether the stored ID and decision match the reply. Hamza ran this check, and 103 of 126 passed.
If the answer requires judgment, such as whether an explanation would mislead the employee, then use an LLM judge. Give the judge the request, expense record, policy, and reply, and have it identify which statements are unsupported by evidence and which evidence conflicts with the conclusion.
The principle is: use the simplest evaluator that solves the problem. Code checks cost almost nothing, can be run on everything after every change, and produce stable results. A judge requires one model call per trace and also needs human labels to validate its judgments first.
The original article's table also includes a typical error. Hamza's first version of the limit check used regex to match wording; later he changed it to judge based on the policy-defined per-night city price, per-person meal cap, and arithmetic. Wording changes; arithmetic does not.
Test the eval first; do not rush to trust the score
A wrong eval is worse than no eval. It tells you a fix worked when it did not.
Testing is not complicated. For each evaluator, keep four small tests: one trace that should fail, one trace that should pass, one paraphrase of each, and one negation. If the evaluator is a model, also add human labels and read its reasoning on cases where you disagree.
For an LLM judge, first have people who understand the policy label traces, with the labelers unable to see the judge's answer. Use disagreements to improve the rubric. After tuning, report two numbers on another set of human-labeled traces: of the cases humans consider failures, how many did the judge catch; of the replies humans accept, how many did the judge let through. You need both. A judge that lets everything through will look great on the second number while missing every failure.
Hamza cites Hamel Husain and Shreya Shankar on binary scoring, as well as Hamel's guide on validating LLM judges.
The replies add a few notes worth recording. One commenter said that checking the eval itself is the step most guides skip, and that playing a few known-bad traces to see whether the evaluator fails is the cheapest approach. Another suggested running the eval against a model that should fail, to confirm that it fails for the right reason. Someone else mentioned paired traces: the same correct action, one citing a real rule and one giving a plausible but wrong reason; if both pass, the judge is not checking evidence.
There is an even sharper comment about Port Selene: a retrieval check cannot catch this error, because the correct paragraph is right there on the screen and the conclusion still used the default cap. The way to test it is to change that city's cap and rerun, then see whether the conclusion follows the document or follows habit.
When changing the Agent, change only one thing at a time
Everything above measures the Agent as it currently is. The payoff from evals appears after changes.
Hamza recommends changing only one thing at a time. The Port Selene trace points to a specific cause: the amount check tool applied a cap to the total, with no unit and no Port Selene rate. So change the tool so it returns the cap, the applicable unit, and the amount calculated per unit. Leave the prompt and model alone.
Before running, write down which cases should flip and which must stay unchanged. In the hotel case group, the case of exactly $250 for one night in a standard city is critical because it can catch a change that breaks the boundary. If the case set has no night in a standard city above $250, a lazy fix that 'raises the lodging cap to $350' will pass every row. That case has to be added first.
Rerunning an Agent has a practical problem. Rerunning a prompt is easy; rerunning an Agent is not. Tools change the world: running again may write duplicate decisions or route reimbursements twice. The world also changes: a record looked up last week may be different this week, and you cannot tell whether the Agent changed or the input changed.
The solution is to replay against recordings. The Agent's real code runs from the beginning, and each tool call is answered with the result recorded the first time. Inputs are fixed, so behavioral differences come from your changes. Kitaru's replay does this. Note that if the thing you changed is a tool, that tool's recorded results are exactly what cannot be replayed; it must run live, while surrounding queries use recorded results. Deciding which tools run live and which are replayed is part of the experimental design.
In Hamza's run, five cases flipped from wrong decisions to correct ones, with no reverse flips. On the finance-record side, two cases were fixed, but four runs replied to the employee without storing the decision. Port Selene already passed in the recorded baseline, so this run by itself cannot prove that the hotel fix worked. Each case was run only once, and among the five flips there could also be normal model variance. It is better to replay each case three times before trusting the result.
To start building your own evals, you need three things
To get started, you need traces, time set aside to read traces, and a habit of asking how each check could be wrong.
The process can be compressed into a checklist: write down who depends on the Agent and what evidence each result leaves behind. Use dimensions to design 100 to 150 cases. Capture complete traces, including the reasons for actions. Read them one by one and write short notes. Group similar notes into five or six failure modes. Write one evaluator for each, using code when the trace has evidence. Give each evaluator one known failure, one known pass, one paraphrase, and one negation. LLM judges need human labels; read the reasoning where there is disagreement. Before changing the Agent, list which cases should flip and which must not move. After replaying, read the regressions first.
This method is not lightweight, and it is not suitable for producing a pretty average score. It is more like reading logs, doing classification, writing checks, and then validating the checks. The benefit is that when the score moves, you know why it moved.
发布时间: 2026-11-21 07:53