Open-Sourcing Hermit: Moving Agent Training from the Model to the Harness

Infini-AI Lab has open-sourced Hermit—a harness evolution framework that lets agents continually learn from task feedback without changing the model or the harness.

Fine-tune the model? No. What changes is the agent’s harness—the logic layer around the model: instructions, tool calls, memory, and sub-agent coordination. The model decides what to say; the harness decides how the work gets done. Most agents do not get stronger from experience; to make them learn something new, you either switch models or risk changing the harness. Hermit offers a third option: train an external adapter.

An adapter is like a plugin with training capabilities that plugs in alongside the agent. It can contain instructions, skills, tools, memory, and coordination logic. During training, the original model and harness are frozen, and only the adapter changes. Once trained, the adapter can be deployed independently and reused across compatible models and harnesses.

Why do we need a framework? Because harness evolution is more than writing prompts. It has to handle task execution, feedback collection, evaluation, update checks, and rollback. If every method builds its own setup, experiments cannot be compared or reproduced.

Hermit provides an RL-style training loop: the acting agent uses the current adapter to perform tasks; task feedback becomes learning material; the Evolver generates the next version of the adapter based on feedback. The Trainer handles execution, feedback, spec checks, version management, budgeting, and recovery. The Evolver only cares about the learning algorithm.

Hermit currently supports GEPA, SkillOpt, MetaWorld Model (MWM), Harness², and Continual Harness; ACE and ReHarness are planned. Each method has its own update logic and evaluation protocol, but they share the same interface.

A concrete example: 5 Codex agents collaborate to prove Lean theorems. The adapter gives them a shared plan, communication channels, and a library of verified lemmas. The agents first break the theorem into lemmas, prove them in parallel, verify with Lean, and put the results into a shared library; other agents pull verified lemmas and keep building on top. Codex itself is unchanged. The code is in the lean_swarm example.

Another example is the OfficeQA benchmark. GEPA and SkillOpt train on 50 questions, select on 24 validation questions, and finally evaluate a fixed adapter. Continual Harness and Harness² learn as they go on a stream of test questions. Evaluation uses GLM 5.3 Flash as the acting agent and Codex CLI 0.162.0.

Some developers asked whether this can be used for enterprise chatbots and financial agents. The author responded: the framework is domain-agnostic; the adapter and acting agent depend only on task feedback. As long as there is feedback, it can be used.

Others worry that it gets better on new tasks but worse on old tasks. The author says it depends on the algorithm design. GEPA and SkillOpt use a held-out validation set to decide whether to accept a new adapter, which can prevent regressions. Continual Harness and Harness² adopt the latest version directly by default. Hermit does not provide a default solution, but it leaves an interface for users to decide whether to use a validation set.

The local walkthrough does not require a GPU, model account, or benchmark download. Run a few commands to see the full training process.

git clone https://github.com/Infini-AI-Lab/hermit.git
cd hermit
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .

In the quick start there is a deterministic example, with baseline 0.0 and final 1.0.

Hermit moves the question of how agents get stronger from experience from the model layer to the harness layer. The artifact is an inspectable, versionable, deployable adapter, and the training infrastructure is managed by a unified framework. The project is open-sourced under Apache-2.0.

GitHub: https://github.com/Infini-AI-Lab/hermit
Docs: https://infini-ai-lab.github.io/hermit/docs/