After Reviewing 16 Eval Frameworks, the Evaluator Is the Biggest Blind Spot
Sergii Makarevych compared 16 eval frameworks and reworked his agent evaluation method. The core issue isn't which tool to choose, but first clarifying which layer you're testing and confirming that your evaluator isn't an uncalibrated ruler.
Sergii Makarevych posted on X saying that after researching eval frameworks, he is restructuring how he builds agents and evals. He attached a mapping diagram that aligns eval types, what is being tested, and references to agent.py files, to show how code should be isolated so evals can run.
He had also written a long article comparing 16 eval frameworks. The core conclusions from that article boil down to a few points:
First, every usable framework forces you through the same four components: data (test cases), target (system under test), evaluation (scorer), and run (save and compare). The differences between frameworks are just the form and depth of these four things.
Second, the evaluator itself is part of the system, not a neutral ruler. The way most teams fool themselves is by treating a high score from an unvalidated judge as evidence. So every eval needs two numbers: how good the product is, and how well the judge agrees with humans.
Third, the layers a framework can evaluate range from granular to broad: final answer, individual step (such as a retrieval or tool call), tool selection itself, the entire path, multi-turn conversation, and cost. Problems at different layers require different frameworks. lm-evaluation-harness is only suitable for running public benchmarks and reproducible model comparisons; Inspect is suited to agent scenarios with tools and sandboxes; Braintrust or MLflow are suited to platforms that need versioned data, release gates, and automatic conversion of production failures into test cases.
Fourth, RAG scenarios should score retrieval and generation separately. Ragas does exactly this, with four metrics: faithfulness, answer relevancy, context precision, and context recall. Only by scoring them separately can you know whether to tune the index or change the prompt.
There are some noteworthy additions in the comments. One commenter said few frameworks let you test the evaluator itself; they specifically generate cheating outputs to fool the judge and see which loopholes slip through. Another commenter pointed out that terms like trajectory and tool-call correctness do not yet have fixed meanings across frameworks: if you run the same trace with the correct answer through two frameworks, they will judge right and wrong in different places. This means the framework you choose actually defines your test.
Someone asked a practical question: among these 16 frameworks, is there any that can turn a bad production trace into a new test case with one click? From the comments, this loop still seems to have to be built by hand; platform-style tools can help, but it isn't a free lunch.
Back to that mapping diagram. When the author was restructuring, the first thing was to clarify the types of evals and what was to be tested, and then isolate the agent's code in a way that can be evaluated. This principle matters more than which framework to choose. Because if you don't know which layer you're testing, no framework can save you.
Original link: How to pick an eval framework that checks what you ship
发布时间: 2026-11-23 05:51