Wink Pings

Trajectories Have Passed 1M Tokens: Judges Should No Longer Read the Full Text

Judgment Labs ran experiments on 100 trajectories ranging from 100K to 2M tokens. Beyond 1M, Agent-as-a-Judge achieved 0.21 higher F1 and 62% lower cost. The questions in the comments are sharper: does the evidence come from the raw trajectory or the agent's own notes, what about injection, and does this still hold for short tasks.

Judgment Labs published a set of evaluation results about how to review long-trajectory agents. The conclusion is not complicated: once a trajectory reaches millions of tokens, making the judge read from beginning to end is both expensive and prone to missing things.

The original article is here: Beyond 1M+ Tokens: New Ways to Evaluate Long-Horizon Agents. The original includes a GIF comparing methods and experimental charts; the link is the same as above.

The evaluation challenge shifts from scoring to finding evidence

Agents are moving from single-turn Q&A to tasks lasting hours. In OpenAI's GDPval, experts take an average of seven hours per task, and some take weeks. In a May 2026 report, METR estimated that Claude Opus 4.6 completes a software task that would take an expert 12 hours with a success rate similar to human experts. Anthropic also has an example of 16 Claude Opus 4.6 agents working in parallel on a restricted C compiler; a human expert working alone would need weeks or months.

As tasks get longer, review gets harder. The original gives an example of migrating customer records: the agent reports that migration is complete, but hundreds of tool calls earlier it merged customers with the same last name, while the user's requirement was to merge only when matched by email. Two different customers are now one record. For the judge to catch this error, it has to connect the instruction at the beginning, the erroneous merge in the middle among hundreds of calls, and the success report at the end.

Evidence may be in only one turn, may span adjacent turns, or may be scattered across the whole trajectory. The third type is the hardest and also the most worth catching. Whether an agent deviated from policy during a long run is often answered by pieces located far apart.

Letting an LLM read everything has direct problems

The common LLM-as-a-Judge approach is to stuff the entire trajectory and the grading rules into the model. Once the trajectory is long, three problems appear at once.

It doesn't fit. On the Judgment Labs platform, 20% of trajectories exceed 1M tokens, and the largest exceeds 1B tokens.

Accuracy drops. Even if the trajectory fits into the context window, model performance degrades before reaching the limit; the original calls this phenomenon context rot.

Cost rises linearly. The judge reads every token, and at production scale cost grows with length.

Two paths: filter chunks first, or let the judge search for itself

The first path is chunk filtering, then handing the remaining content to an LLM judge. Chunks overlap so evidence is not cut at boundaries. There are two kinds of filter: a small model extracts verbatim quotes chunk by chunk, and a decision model, Jev, labels each chunk as relevant, irrelevant, or uncertain.

The second path is Agent-as-a-Judge. It treats evaluation as a search problem and uses tools to find evidence in the trajectory. The trajectory is exposed as a SQL table, spans, one row per turn; the agent writes SQL queries and can also full-text search test names or error messages. Heavy work runs in a stateless execution service, and intermediate results are stored in a key-value store. Each turn carries metadata such as type, turn number, parent node, and tool name, so the agent can pre-filter.

The comparison diagram in the original shows that the chunking method reads one chunk at a time and then hands the remaining content to an LLM judge; Agent-as-a-Judge directly searches the full trajectory for evidence and makes a judgment.

The numbers: the gap widens on long trajectories

The experiment used 100 trajectories: 20 each at 100K, 500K, 1M, 1.5M, and 2M. There were two grading rules: find all hallucinations, and find every step where the agent skipped a skill. Scoring checked whether the turns cited by the judge matched pre-inserted failures, measured with F1.

Averaged across four models, within 1M tokens, Agent-as-a-Judge had evidence F1 of 0.85, LLM-as-a-Judge had 0.70, and Jev to LLM-as-a-Judge had 0.66; the Agent version cost 42% less. Beyond 1M tokens, the gap widened: the Agent version was 0.21 F1 higher than LLM-as-a-Judge, 0.27 higher than the Jev route, and 62% cheaper.

At 2M tokens, Agent-as-a-Judge averaged 0.80 F1 at $0.49 per evaluation; LLM-as-a-Judge was 0.60 F1 at $1.48. DeepSeek V4.1 running the Agent version maintained 0.94 to 0.995 F1 from 100K to 2M. On cost, at 2M the Agent version of GPT-6 Luna was $0.10 and the LLM version was $0.71; for GLM 5.3 they were $1.26 and $4.03 respectively.

The flaw of chunk filtering is its locality

Chunking methods can get the judge past the context window, but each chunk only sees itself. The two halves of a failure may be 230 turns apart. The report says all tests passed, while a test 230 turns earlier actually failed. Read only the report chunk, and it looks like an ordinary summary; read only the test-failure chunk, and it looks like the agent may have fixed it later. Each half looks harmless alone, so the filter may miss one side.

The Jev route also retains too much content. The original says it kept 77% of chunks containing failures and also kept 36% of chunks without failures, cutting away only 29% to 45% of the trajectory. In the end the LLM still has to read most of the content, and recall at 2M shows no clear improvement.

Agent-as-a-Judge works differently. When it reads a report saying tests passed, it searches test runs, no matter how far away that segment is. The two halves of the failure are viewed together. The original's conclusion is direct: every problem comes from one design choice, the judge read the whole trajectory. Most trajectories are not relevant to a given grading rule.

The questions in the comments are more worth reading than the leaderboard

A commenter, John Rood, said the winner is the judge with hands, not the one with a bigger window. His second half is sharper: an LLM judge judges the agent's narrative of its own work, and that is the version the agent already believes.

Karan Jagtiani wanted a number: how much of the judge's evidence comes from the raw trajectory, and how much comes from the agent's own progress notes. A judge that reads notes inherits what the agent omitted. Exori asked about the security boundary: any line in the trajectory might speak to the grader; have injection tests been done. Jerry Liu asked, if beyond 1M the judge is mainly retrieving, then compared with a chunked LLM judge that has a retrieval step, is the 0.21 gap only valid for ordinary chunking.

These doubts should not be covered up by F1. Catman also reminded people that chunked judges are still practical for short tasks, so don't treat the long-trajectory result as a total victory.

Long-trajectory evaluation looks more and more like search

Judgment Labs' conclusion points to an engineering reality: grading rules need to be specific enough to determine what counts as evidence; the judge needs to be able to search the full text and semantically similar content; sub-agents can search in shards; a decision model can combine scores; and the judge should also be able to inspect environment state changed by the agent, not just the trajectory text.

Someone in the comments summed up reality in one sentence: in a billion-token trajectory, the bug is still at step 4. The longer the trajectory, the more the judge must not only read, but also find.

发布时间: 2026-11-18 18:33