Your AI Pass Rate Might Just Be the Grader Nodding Along

Sergii Makarevych wrote an evals checklist for engineers, product managers, and CEOs. It puts the step most often skipped in AI product evaluation first: prove that your grader is trustworthy.

Put simply, an eval is a repeatable test: give the system a batch of inputs, compare the outputs with the reference answers, and end up with a number that has an error range. The original article uses a bicycle shop's customer service assistant as an example: 80 tickets, 20 used to tune prompts and the grader, 60 used only for reporting. Triage misclassification, replies missing facts, invented policies, and a refund agent changing the database incorrectly are each scored separately.

These four steps sound ordinary; the traps are in the details. Inputs need to look like real traffic, including angry customers, ambiguous questions, and multi-question tickets. Next to each input, you need to write clearly what correct looks like: the classification label, which section of the manual must be cited, and which two or three facts must be stated. The original article makes a hard judgment: inputs with reference answers are the most expensive asset in the whole process; frameworks and dashboards come later.

Read outputs before writing graders. The original team counted 60 test replies: 34 passed, 15 lacked required facts, 12 made unsupported claims, 8 retrieved the wrong section, and 5 failed to answer the question. This failure list becomes the specification for all later graders. The first week's work is unglamorous: read one to two hundred outputs, write failure notes for each one, then group and count them. Skip this step, and teams easily write graders for failures that do not exist, producing a reassuring number.

Code graders are cheap, fast, and stable, and they suit checks with definite answers: classification labels, word count, prohibited promises, and whether required facts appear. Their blind spots are also clear. In the original article, only 11.7% of replies passed all code rules, but the code rules did not catch invented claims. A customer service reply invented "a total allowance of up to 15 business days"; format, citations, and required facts all passed, and only the model judge found the problem. Rules see form, not meaning.

Model judges can read semantics, but they make mistakes too. The original article calibrated the judge with 60 manually labeled replies: raw agreement was 60%, which looks okay; Cohen's kappa was only 0.15. A judge that says pass to every reply can also get 57% raw agreement, because most replies pass anyway. More troublesome are the judge's known biases: favoring the first answer that appears, favoring long answers, favoring its own model's style, and easily mistaking a pleasant conversation for task completion. The mitigations are not complicated: swap order in pairwise comparisons, use binary questions with evidence, use judges from different model families, and show it tool calls and final state together. First test the judge on a few dozen human-labeled examples, report kappa or the bad-answer catch rate, and only then let it participate in release decisions.

Numbers need error bars. The original article's triage accuracy is 0.700, with a 95% confidence interval [0.583, 0.817]. The reply pass rate for both versions is 0.700, the difference is +0.000, interval [-0.117, +0.117], and the conclusion is that there is no measurable difference. The detection floor for a 60-item test set is about 12 percentage points; a 5-point regression is invisible at this sample size. There is another often-ignored account: the overall pass rate staying the same does not mean no customers got worse. The original v2 regressed on 14% of cases while improving on others. Before release, you should ask which cases got worse, not just look at the total score.

Agents need to be evaluated separately. Is the final database state correct, are the paths allowed, does the task succeed all three times when run three times? The original article uses pass^k to measure reliability and pass@k to measure capability. In customer service scenarios, pass^3 is closer to what users experience than succeeding once. Cost also has to be counted: token consumption for agentic tasks can be a thousand times that of a chat turn, and the same task with the same model can vary 30x across runs. Ranking models by cost per correct result is closer to the business than looking only at accuracy.

Public benchmarks are for screening engines; they are not responsible for proving your product can ship. With the same model, changing the prompt format can move GSM8K from 0.686 to 0.746; benchmarks get contaminated and saturate, and the top of the leaderboard is often within noise. The original article's advice is to use five benchmarks to narrow the candidates, then decide with your own eval.

The rollout cadence can be quite plain: run code checks and format checks on every commit; every time you change prompts, models, or retrieval, run the model judge on the test set and sample human review; after launch, sample real traffic, run cheap graders on everything and expensive graders on slices. State the detection floor in the CI gate; don't describe a 3-point gate as if it can catch a 5-point regression.

Some additions in the comments are even more stinging than the original article. One commenter said a readable pass rate is not the same as a release gate; you need to write down which set, what score, and who calls a stop when the grader lets bad answers through. Another reminded readers to test the judge with known answers first; if the judge cannot even handle easy questions reliably, everything after that is noise. Another added an engineering detail: when the judge call fails, many frameworks simply drop that item from the average, and the number still looks good. Sergii Makarevych's checklist is already long enough; these comments fill in the accountability gap inside organizations.

Source: Sergii Makarevych original article. The checklist in the image also comes from the same article.