Wink Pings

The biggest pitfall of eval frameworks: You've never checked the scorer

A scorer is part of the system, not a neutral yardstick. The author reviews 16 eval frameworks to help you avoid selection pitfalls: define the layer first, then choose a framework, and finally calibrate your judge.

Language models fail fluently: their answers are confident and well-structured, but may be wrong, unsafe, or fabricated out of thin air. Reading one correct answer won't reveal the problem; the next one may go off the rails. An eval framework turns this worry into a loop: collect inputs, run the system, score, and save comparisons.

But before choosing a framework, there is a more fundamental question: which layer are you testing? The final answer, a single step, tool calls, the entire trajectory, multi-turn conversations, and cost are different checks. Choose the wrong layer, and everything after it will be wrong.

Sergii Makarevych reviewed 16 eval frameworks in a long article. His core warning is: a scorer is part of the system, not a neutral yardstick. Every eval needs two numbers: how good the product is, and the agreement rate between the scorer and humans. The most common way to fool yourself is to trust an uncalibrated judge.

Four basic components

Almost every framework revolves around these four things:

  • Data: test cases, input + expected answer + context
  • Target: the system under test—model, prompt + model, chain, agent, retriever
  • Evaluation: scorer, code rules, model judge, decision model, human
  • Run: save comparisons, experiments, logs, matrices

Six levels

The author groups the 16 frameworks into six levels.

Level 1: Shared shape

Every framework's eval looks the same: data, target, evaluation, run. Different names, same essence.

Level 2: Code-first

The representative is Inspect. It implements the agent solver as a loop: model calls, tools, file reads, sandbox, memory, subagents. The sandbox lets code written by the model run in an isolated environment that doesn't touch your computer. Scoring can be regex, similarity, model judge, or custom code.

An Inspect code example:

from inspect_ai import Task, task
from inspect_ai.dataset import example_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate, system_message

@task
def security_guide(system="devops.txt", grader="expert.txt"):
    return Task(
        dataset=example_dataset("security_guide"),
        solver=[system_message(system), generate()],
        scorer=model_graded_fact(template=grader)
    )

DeepEval turns pytest into a runner, running evals in CI like unit tests. It has many ready-made metrics, but trajectory metrics require its own tracing. Pydantic Evals uses typed cases and reports, supports an independent judge for each case, and can also sample online. There is also an emerging SkillEvaluator, specifically for testing the loading, overlap, and behavioral improvement of the skill folder itself, with about 4,000 stars, but its support is still experimental.

Level 3: Config-first

lm-evaluation-harness uses YAML to define tasks and runs public benchmarks, with stability relying on seeds and repeats. Promptfoo uses YAML matrices to test prompt × provider × case, with rich assertion types, including tool calls and RAG, making it suitable for fast regression.

Level 4: Platforms

Braintrust treats experiments as products, with immutable datasets, gates written into CI, and online rules with sampling. MLflow GenAI is a self-hosted alternative, with a larger judge catalog and pytest for gates.

Level 5: Observability

Langfuse attaches evaluators to traces and datasets, creating an online + offline closed loop, with cost control. LangSmith suits the LangChain stack but is hosted-only. Phoenix uses pure-function evaluators, with prebuilt path templates. TruLens's RAG trio and judge calibration are selling points. Evidently does table row-level checks and is suitable for tabular data.

Level 6: Single-purpose metrics

Ragas does one thing: it scores retrieval and generation separately. Four core metrics: faithfulness, answer relevancy, context precision, context recall. Note that context recall requires a reference. Ragas itself does not provide history or gates; it must be embedded in another loop.

Quick selection overview

LevelRepresentative frameworksSuitable for
Code-firstInspect, DeepEval, Pydantic EvalsAgents with tools that need sandboxing or pytest gates
Config-firstlm-harness, PromptfooBenchmark comparison, fast regression
PlatformsBraintrust, MLflow GenAINeed versioned experiments and gates
ObservabilityLangfuse, LangSmith, PhoenixDebugging online paths
RAG metricsRagasOnly tuning retrieval or generation

A few action rules

  • Define the layer first, then choose the framework.
  • The scorer must have its own score. Measure its agreement with humans first.
  • Code-first suits agents with tools and state; config-first suits benchmark numbers and fast matrices.
  • Platforms buy you versioned data, comparisons, gates, and an online closed loop, but they cannot buy you statistical repeats.
  • Observability tools buy debugging capability, but they require explicit judges and thresholds to make claims with confidence.
  • RAG metrics tell you where the problem is: low precision means fix retrieval; low faithfulness means fix generation.

Source

Based on Sergii Makarevych's article 'How to pick an eval framework that checks what you ship', original link: https://x.com/i/article/2124208252066152448 . The author's view is: the value of an eval framework is not in the number of metrics, but in whether it can turn failures into datasets, and then turn datasets into gates.

发布时间: 2026-11-22 05:40