Wink Pings

Eval Optimizer: Agent evaluations don’t need a frontier model every time, cutting costs by 50x

Judgment Labs has released Eval Optimizer, which automatically allocates the right amount of compute for each evaluation. The company says that in one customer case, the cost per evaluation dropped from about $0.06 to $0.0016, a 30–50x reduction.

There is a hidden waste in agent data evaluation: running every check through the same top-tier model. Some questions can actually be decided with a few lines of code, some need a lightweight classifier, and only a few require full long-context reasoning.

Judgment Labs today released a tool called Eval Optimizer, positioned as a “router”: you describe what you want to evaluate, and it automatically selects the judge—i.e., the evaluation model—and configures the evaluation pipeline. Here, a judge can be deterministic code, a decision model, an LLM, or even a full Agent. In the customer case shared in the official blog, the cost per evaluation dropped from about $0.045–$0.075 to $0.0016, a 30–50x reduction.

Why evaluation is so cumbersome

Agent data evaluation looks complex, but at its core it is the same thing: have a model or piece of code read through a trajectory and then assign labels. A trajectory is the record of an Agent’s interactions with the outside world. Evaluation can happen during real-time monitoring, offline testing before launch, or post-hoc data analysis. The difference lies in when it runs and which data it runs on.

Every evaluation first requires configuring a judge. This configuration process is currently very manual: you have to read trajectories, define a rubric (what counts as passing and what counts as failing), set the scope (whether to look at a single turn or the entire session), and then choose the model and amount of compute. Rubrics usually need several back-and-forth iterations before they align with human preferences.

What Eval Optimizer does is automate as much of this configuration as possible. It reads trajectories, finds where the evidence for each rubric item is concentrated, and then decides which tier of model to use. To determine “whether the Agent called the complete_task tool,” a few lines of deterministic code are enough; there’s no need to run a frontier model on an 8×H200 node. To determine “whether the user is a bit frustrated,” a decision model may suffice. Only when the evidence requires large context, the judgment requires real reasoning, or external environments (such as code repositories or databases) must be checked does it make sense to use an LLM or a full Agent.

This partitioning logic can be drawn as a chart: the horizontal axis is trajectory span, and the vertical axis is rubric difficulty. Deterministic code covers programmatic checks, decision models cover small-context classification, LLMs handle cases requiring reasoning or explanation, and Agents handle long trajectories plus external verification. The boundaries are gray areas, and also the hardest part to determine.

Here is an official demo video showing the process from user description to judge creation:

Customer case: 30–50x cost reduction

The official blog shared a customer case. This customer was running narrow questions: the result of a single turn and the type of the user’s first query. Their trajectories averaged about 1 MB. If an LLM read the full trajectory, a single evaluation was estimated at $0.045–$0.075. Eval Optimizer routed these questions to a decision model and used code preprocessing to extract only the evidence the decision model needed, bringing a single evaluation down to $0.0016. The official post says this is not even the most extreme configuration.

Some commenters pointed out that cheap judges may drift as Agent behavior changes. The solution is to periodically re-score a small random sample with an expensive judge and keep an eye on it. Other commenters worried about whether scores across different evaluations are comparable, and suggested keeping a fixed frontier model as a reference. These are all issues worth noting in real-world use.

What’s next

The official post lists three directions: first, publish an in-depth technical article on how preprocessing code is generated and how to run it safely at scale; second, custom inference for evaluation—once the judge is configured, the workload is known, so work can be shared and throughput optimized; third, fine-tune small decision models using customer-labeled trajectories to make judges more accurate and cheaper.

The most notable point of this release is: evaluation should not be priced by model size, but by problem difficulty. Not every Agent evaluation deserves a frontier model, but not every problem can be dismissed with a few lines of code either. Eval Optimizer sits right in the middle, turning “which model to use” into a choice rather than a default.

发布时间: 2026-11-20 07:49