A leaderboard you can't reproduce is just a claim: Parallel open-sources its search evaluation tool
Every few weeks there's a new leaderboard, and its own bar is the tallest. Parallel has open-sourced parallel-search-evals, which supports bringing your own questions and letting a coding agent run evaluations.
Parallel has open-sourced parallel-search-evals. The repository is used to measure the performance of Parallel Search, to evaluate new models deployed internally, and to support the Search Capability Leaderboard. It's at parallel-search-evals.
Search evaluation has long had a transparency problem. Every few weeks, an AI provider publishes a new chart with its own bar the tallest and competitors a notch shorter. Readers see only the results, not the model configuration, tool settings, error logs, or discarded failure cases. In an article, the Parallel team calls this out directly: benchmarks that can't be reproduced are just claims. That's not a new point, but they've released the tool so teams can first reproduce Parallel's results and then run experiments on their own data.
The harness supports any large model accessible through OpenRouter, OpenAI, Anthropic, Google, or an OpenAI-compatible endpoint. Each run evaluates one model, one benchmark, and one mode. Modes include turbo, fast, basic, advanced, and baseline. Baseline is closed-book with no tools; comparing baseline with search runs shows the improvement from search.
Using it isn't complicated. First clone the repository and sync dependencies:
git clone https://github.com/parallel-web/parallel-search-evals.git
cd parallel-search-evals
uv sync --frozen
Three-question smoke test:
uv run --frozen parallel-search-evals run --provider anthropic --model claude-sonnet-5 --benchmark dsqa --mode fast --limit 3 --output runs/dsqa-smoke
Public benchmarks are only a starting point. The decision data that really matters is in your own traffic. The harness separates the question set from the agent loop and grader, so teams can add their own questions, reference answers, and scoring rules while keeping search configuration, budget, and reporting aligned with the leaderboard. The repository also includes an agent skill: .agents/skills/run-search-evals. Point Claude Code or Codex at the checkout and have it run evaluations; it will handle models, modes, question counts, credential checks, runs, resumption, and summaries.
Mechanically, the harness handles things general-purpose frameworks don't usually cover: multi-turn tool calls using native protocols across providers, per-call cost accounting, and recovery after a provider stream drops mid-run. It integrates with OpenRouter to simplify multi-model access and also keeps native adapters for OpenAI, Anthropic, and Google, including Anthropic prompt caching and Gemini continuation tokens.
Parallel provides two tools. The Search API accepts a natural-language goal and short keyword queries and returns up to 10 results with dense excerpts. The Extract API pulls evidence from up to 20 specified URLs. Models get a budget of 20 tool calls per question. Deep research mode gives the model two tools and requires it to forget its internal knowledge and use only those tools; after the budget is exhausted, it answers based on the information it has found. An OpenAI GPT-6 Sol judge scores with the benchmark's official prompt: DeepSearchQA uses F1, allowing partial credit for incomplete answer sets, and HLE uses strict correctness.
If you find a bug or disagree with a design decision, open an issue on GitHub. The original tweet is here, and the article is here. Leaderboards are worth looking at, but run them once before you believe them.
发布时间: 2026-11-18 09:27