47/100: Web agents' real scores are first cut in half by the grader

Browser Use and CMU have released Odysseys2. It has 100 tasks drawn from the distribution of real Browser Use user requests, anonymized, rewritten, and expanded, with both tasks and scoring criteria reviewed by humans. The tasks run on the live web with a 60-minute limit, and each task has 8 to 12 scoring criteria, 982 in total. There are 15 agents and 3 harnesses. The strongest, GPT-6 Astra, averages 84.6 and perfectly completes 47/100. Five agents do not perfectly complete even one task.

The tweet says, 'Web agents are becoming better than humans at knowledge work.' Someone in the comments replies: SOTA only completes 47/100, and five agents complete zero, but of course it is better than humans. The contrast is not hard to read. Web agents for long tasks are still far from reliable delivery; don't rush to declare that they will replace knowledge work.

Why do the numbers suddenly look bad? The old Odysseys judge only looked at action history and screenshots. Many tasks require delivering spreadsheets, reports, or files, but the judge did not get the file contents. The Odysseys2 judge looks at delivered file contents, action history, and screenshots together, judges all scoring criteria together, requires evidence to be written out, and assigns 0 for suspected fabrication and reward hacking. On the same batch of 2,098 attempts, the old judge averaged 88.9 and the new judge 45.3. Gemini 3.7 Flash drops from 94.9 to 5.8. With high effort, Gemini 3.8 Flash has 75/100 attempts scored 0.

In a reply, Sachin Mour says the old judge never received the files the agent delivered, while the new judge opens them. Task 002 is an example. Gemini 3.8 Flash was counting authors of arXiv papers; at step 108 of the log it explicitly says the exact phrase 'privacy in machine learning' does not exist, yet the final report still counts two papers. The Odysseys2 judge scores it 0 for suspected fabrication. Once the grader can read the work, errors can no longer hide outside the screenshots.

Human review is not easy either. In the Prolific pilot, there were 5 tasks, with two reviews per task, $5 and an expected 20 minutes. The actual median was 49 minutes, and 29 reviews were not submitted. Of 75 ratings, 0 were 'not met.' In Task 005's CSV, one row had a Google Sites address ending in /edit; both human reviewers and the old judge marked it as met, while the Odysseys2 judge caught it from the run log. Opus 5.5 reviewed the same materials in 2 to 5 minutes per review and caught it both times. Logs for long trajectories are often 16,000 to 41,000 words, so manually checking item by item is not cheap. This pilot is very small; it can only show that human review of long trajectories is hard, not that the judge is accurate overall.

Beyond the grader, the harness also changes scores. The same GPT-5.6 Luna scores about 33 with the Python library, while BrowserCode and Pi at high effort reach 59 to 72. Choosing the harness is as important as choosing the model.

Speed and cost must also be considered together. At 15 minutes, Luna scores 37 and Opus 5.5 scores 18. At 36 minutes, Opus catches up. At 60 minutes, Opus scores 71 and Luna 60. Luna costs $0.35 per task, Opus $6.22. For speed and low cost, Luna has the advantage; for final quality, Opus overtakes after a long wait.

Single-run scores fluctuate a lot. For 8 runs of Luna with high effort on BrowserCode over 100 tasks: average score 56.9, best@8 83.1, worst@8 23.1. Average perfect rate 6.25%. 26/100 tasks are perfect at least once, and 0 tasks are perfect all 8 times. Treating one run as stable capability will overestimate it.

Task distribution: shopping 27%, research 20%, finance/legal 14%, catalogs 12%, media/games 10%, real estate 9%, local travel 8%. The median instruction is 271 words and usually requires multiple deliverables. For example, Task 029 asks to trace the Брук family in an 1882 handwritten register, requiring OCR from page 9 to the last page, marking readability page by page, preserving conflicting spellings, ages, and places, giving confidence levels, and not logging in, editing, uploading, or contacting anyone. This kind of task is not a toy.

Odysseys2 open-sources the tasks, scoring criteria, scoring prompt, and scores under MIT. Download links: tasks, evaluation rules, judge prompt, benchmark repo. Evals are now the bottleneck for agents. Scores went from 88.9 to 45.3, but the agents did not suddenly become dumber. The grader started reading the work.