What agent evaluation lacks is not a leaderboard, but a lab you can rerun
Harbor comes from the Terminal-Bench team. It runs agents such as Claude Code, OpenHands, and Codex CLI in real containers, supports parallel execution across thousands of environments in local or cloud sandboxes, and connects the agent, model, dataset, policy, sandbox, verifier, and artifacts into a reproducible closed loop. For agent builders, it adds something beyond the demo that they can inspect.
Agent evaluation currently has two common scenarios: comparing scores on leaderboards, and checking results in demos. Harbor aims to add a third: turning evaluation into experiments that can be rerun.
Fajar M Reza introduces in a tweet that Harbor is a framework for evaluating and optimizing agents and language models in real container environments. The project comes from the creators of Terminal-Bench and is licensed under Apache-2.0. It supports agents including Claude Code, OpenHands, and Codex CLI, and can also run versioned benchmarks.
A leaderboard gives you a screenshot of rankings. Harbor gives you an experiment that can be rerun and whose failures can be inspected. Fajar M Reza focuses the useful part on the complete loop: agent, model, dataset, policy, sandbox, verifier, and artifacts, with concurrency and reproducible runs exposed through a CLI. The codebase also exposes Harbor operations as MCP tools so other agents can call them.
For agent builders, eval researchers, and platform teams, this can turn “the demo looks good” into reproducible, inspectable experiments. According to the tweet, this is where Harbor’s value lies.
Installation:
uv tool install harbor
Or:
pip install harbor
Harbor is the official harness for Terminal-Bench-2.0. The command to run Terminal-Bench-2.0 is:
harbor run --dataset terminal-bench@2.0 --agent claude-code --model anthropic/claude-opus-5-5 --n-concurrent 4
By default, it starts locally with Docker. To switch to a cloud sandbox, add --env daytona and configure DAYTONA_API_KEY:
harbor run --dataset terminal-bench@2.0 --agent claude-code --model anthropic/claude-opus-5-5 --n-concurrent 100 --env daytona
It can run thousands of environments in parallel through providers such as Daytona, Modal, and LangSmith. harbor datasets list shows supported third-party benchmarks, including SWE-Bench and Aider Polyglot. Harbor can also generate rollouts for RL optimization.
Project address: harbor-framework/harbor. End-to-end examples and guides are in the Harbor Cookbook. If you want to cite it in academic work, the GitHub page provides BibTeX and a DOI.
Judging an evaluation framework is not just about whether it can run a few more benchmarks. Leaderboards spread quickly; reproducible experiments get no retweets. For teams building agents, the latter is more valuable. Harbor’s test lies elsewhere: whether environments can reproduce reliably, whether verifiers can cover real failures, and whether the cost and scheduling of cloud sandboxes can sustain thousands of concurrent runs. These problems won’t disappear because of a CLI. Start with the end-to-end examples in the Cookbook, then decide whether to move your own benchmark into it.
发布时间: 2026-11-05 06:27