Wink Pings

Parallel Open-Sources Search Evaluation Tool: A Leaderboard That Can't Be Reproduced Is Just a Poster

Parallel has open-sourced parallel-search-evals. The model, tool budget, grading rules, and smoke test are all laid bare. Whether the leaderboard can be trusted—run it and see.

Parallel has open-sourced parallel-search-evals. It is a search evaluation toolchain used to measure the performance of Parallel Search, evaluate new models deployed internally, and support the Search Capability Leaderboard.

Leaderboards have an old problem. Every few weeks, new charts come out: the publisher's bar is the tallest, and competitors' bars are a notch shorter. Readers can see only the results, not the model configuration, tool settings, error logs, or failed samples. If the results can't be reproduced, the leaderboard is just a poster.

This harness is meant to solve exactly that. It lays the evaluation process open, so teams can first reproduce Parallel's results and then swap in their own datasets to experiment.

The repo is here: parallel-search-evals. The original tweet is here, and Parallel's article is here.

It supports models accessed via OpenRouter, OpenAI, Anthropic, Google, or any OpenAI-compatible endpoint. Each run tests only one model, one benchmark, and one mode. Modes include turbo, fast, basic, advanced, and baseline. Baseline is closed-book, with no tools provided. Put the baseline and the search run side by side, and the difference is the gain from search.

The getting-started commands are not complicated:

git clone https://github.com/parallel-web/parallel-search-evals.git
cd parallel-search-evals
uv sync --frozen


# Three-question smoke test
uv run --frozen parallel-search-evals run \
  --provider anthropic --model claude-sonnet-5 \
  --benchmark dsqa --mode fast --limit 3 --output runs/dsqa-smoke

This harness can also be adapted into a tool for comparing Parallel with other providers. Public benchmarks are only the starting point. The decisions that really need to be made are hidden in your own traffic. It separates the question set, agent loop, and grader. You can add your own questions, reference answers, and grading rules, while search settings, budget, and reporting stay the same.

The repo also includes an agent skill at .agents/skills/run-search-evals. It has the coding agent confirm the model, mode, number of questions, and credentials step by step, then run, resume, and summarize. Point Claude Code or Codex at the checkout and let it run the evaluation.

Search evaluation has a few things that existing frameworks don't handle well: multi-turn tool calls across providers' native protocols, per-call cost accounting, and recovery after a stream drops midway through a long research chain. OpenRouter connects to hundreds of models through one integration, which is convenient. The native OpenAI, Anthropic, and Google adapters preserve each provider's own tool-calling and reasoning protocols, including Anthropic's prompt caching and Gemini's continuation tokens.

Parallel provides two tools. The Search API takes a natural-language goal and short keyword queries, and returns up to 10 results with dense excerpts. The Extract API pulls evidence from up to 20 specified URLs. The model has a budget of 20 tool calls per question. The evaluation runs in deep research mode: the model gets these two tools and is told to forget its internal knowledge and use only the tools to find answers. When the budget runs out, it must return a result based on what it has already found.

Grading uses an OpenAI GPT-6 Sol judge, comparing against reference answers using each benchmark's official prompt. DeepSearchQA uses F1, and incomplete answers receive partial credit. HLE uses strict correctness. The component list includes Models, Web search tools, Tool budget, and Grading.

Under the open-source announcement, the comments were lively. Someone shouted, "open source for the win." Kujo posted this image:

Kujo 在回复里贴的图

Others asked more practical questions. DataMining asked whether the repo includes a grader or only a runner. Milhous wanted to see scores for completed tasks side by side with scores that include failed tasks, to judge whether a low score meant wrong answers or tasks that didn't finish. Lachy asked whether the pass/fail threshold for new model evaluations is fixed in advance or readjusted after each model change. The original text did not answer these questions one by one.

These follow-ups are closer to everyday use than the bars on a leaderboard. Whether an evaluation tool is good depends on whether it can be rerun, whether it can be plugged into your own question set, and whether it leaves inspectable traces when things fail. Parallel has open-sourced the tool; the next step is getting more people to run it and produce different results.

发布时间: 2026-11-19 01:36