When the Reward Function Disappears, the Environment Becomes the Reward: Hugging Face's Agent Training Series Concludes

On October 26, Hugging Face held an in-person event in San Francisco to wrap up its six-month Training Agents series. Four talks plus one roundtable focused on the same thing: post-training agents with open models in industrial environments.

The full recording is here, and the slides and code are also public:

The series itself is a progressive tutorial. Six livestreams went from evaluation all the way to environments:

1. Agentic Evaluations Workshop: Discussed the state of agent evaluation, and why benchmark scores don't line up with real-world experience. Participants included Arvind Narayanan of Princeton, Pierre Andrews of Meta, and others.
2. RL for Agents Workshop: Environments, rollouts, reward design, and the reasoning bottleneck that appears when moving from RL for LLMs to RL for agents.
3. Training Agents 1: SFT on agent traces: Used public coding-agent traces to do TRL + LoRA fine-tuning on Hugging Face Jobs. It also honestly discussed what the first round of eval numbers can and cannot tell you.
4. Training Agents 2: Distillation: off-policy, on-policy, and self-distillation, distilling larger-model capabilities into smaller coding agents.
5. Training Agents 3: Reinforcement Learning: Ran GRPO after SFT, including group sampling, verifiable rewards, and reading reward/KL/length curves. One experiment deliberately designed a hackable reward, letting the agent hack the reward during training; the effect was very intuitive.
6. Training Agents 4: From reward functions to environments: Rewards shifted from functions to the space in which the agent acts. From Gym's reset()/step() contract to LLM agents, they built an OpenEnv environment live, pushed it to the Hub, connected it to TRL's GRPOTrainer, and finally trained a real coding agent (OpenCode) on Hugging Face Sandbox via Harbor.

The six livestreams have accumulated more than 300,000 views. For content with hands-on steps, code, and recorded replays, this number shows that people really are looking for a methodology for "how to train your own agent."

The in-person wrap-up event brought in industry practitioners:

• Gobind (Microsoft, GitHub Foundry): Enterprise RL experience. Most of the customers he serves are in traditional industry, manufacturing, and energy, not Bay Area startups. Customers don't care about the model itself; they care about latency, accuracy, and de-risking. The entire system (tools, data, network, harness) affects the final result. He recommends starting from an existing harness, collecting OTE traces from production environments, and then constructing the RL environment. They should also deliberately inject stale data, timeouts, and other conditions into the environment to simulate real-world perturbations.
• Victor (Mercor): Environment design.
• Miles / Tao Lin (RadixArk): How to scale agent RL.
• Xiangyi Li (BenchFlow): Why build OpenEnv Arena.

One training result worth highlighting separately came from the sixth session. The model is sergiopaniego/qwen3-1.7b-mbpp-grpo, using Qwen3-1.7B to run GRPO in OpenEnv's code environment. There is no reward_funcs in the training script—the environment itself handles the reward, scoring by running the task's hidden tests and returning the pass ratio.

On the 257 problems in MBPP sanitized/test (unseen during training), the base model Qwen3-1.7B scored 0.518, and this model scored 0.582. A more reliable metric is the eval/reward curve: it improved from 0.541 to 0.635, with a t-value of 5.18. But train/reward looks almost flat, because each step samples only 4 problems; the difficulty variation among MBPP problems is far larger than the single-step improvement, and the noise of 0.213 overwhelms the effect of 0.06. The model card puts it directly: "Watch eval/reward." This is a common trap in RL training.

Someone in the comments asked: beyond the reward function used during training, how does this agent perform? Ben replied that it improves on a held-out task set in the same environment, but admitted that ideally a benchmark should also be run. Another person said, "Astra can do everything, do we still need this?" Ben's reply was very down-to-earth: "You can train an agent to run on your own laptop, for free, or to run in a toaster."

All the code, environments, and model cards from the entire series have been open-sourced. If you are planning to train your own agent, starting from that playlist is much easier than starting from papers.

Playlist: https://www.youtube.com/playlist?list=PLo2EIpI_JMQvQZm-kVlz4wY1vWF0LBcf5