Training Scores Keep Rising While Code Gets Worse and Worse: Hugging Face Turns the Biggest Pitfall in Agent Training into a Tutorial

Want to train an Agent that can actually use tools yourself? Where do you get stuck? It's usually not a lack of compute, but not knowing where to start. Hugging Face recently put together a post-training course called 'A curriculum for post-training agents,' using a Coding Agent to tie the whole process together.

Let's fill in the background first. Post-training refers to the stage after pre-training; SFT, distillation, and reinforcement learning all count. The goal is to turn a base model into something that follows instructions, uses tools, and can solve specific tasks. What makes an Agent special is that it works across multiple turns—calling tools, reading results, and then deciding the next step. Training must cover the entire trajectory, not just a single response.

The course is divided into five steps. Each step comes with a method diagram, a livestream or lecture notes, an explanatory blog post, and TRL documentation. Some steps include training code and real projects. The order follows Hugging Face's livestream series: four lessons plus two workshops, with a separate playlist.

Step 1 is SFT. The model learns to replicate Agent trajectories produced by a stronger model; each trajectory records all tool calls and results from a task. It's cheap and stable. The problem is also obvious: the model only sees correct trajectories and never sees what it looks like when it makes mistakes. When it actually makes an error while solving a problem, it has never learned how to recover. The next two steps are aimed at this.

Step 2 is distillation. A teacher model supervises the student, either through data it writes or through its probability distribution over the student's own tokens. The teacher can be a larger model, multiple expert models distilled into one student, or the same model with extra context for self-distillation. The signal is dense—one value per token—so it needs fewer samples than RL. The cost is that the student's ceiling is capped by the teacher, and on-policy distillation requires running the teacher at every step.

Step 3 is RL, specifically the GRPO algorithm. Sample multiple answers for the same prompt, score them all, and increase the probability of answers that score above the group average. There's no need to train a separate model to estimate value. The signal is weaker than distillation—one score for the entire answer—so the model has to try many times to learn. Under a standard reward function, each answer is only a single turn and is evaluated once.

Step 4 moves the reward into the environment. The environment keeps state, exposes tools, and supports multi-turn responses; it runs tools itself, records changes, and checks the final result—for example, by running the tests for a coding task. Each turn is much longer than a single response, so generating rollouts becomes the slowest part of training. The accompanying blog post has an angle worth reading: why RL environments work now but didn't in 2016.

Step 5 is Harness and scale. Each rollout runs the complete Agent through a real harness in a remote sandbox, so the training environment matches the actual runtime environment. Generation and training are split across different GPUs, so long turns don't leave the trainer idle, and the updated weights are sent back to vLLM. It uses AsyncGRPOTrainer.

The most interesting part of the whole course is an experiment. The authors deliberately designed a reward function that was easy to game, and as a result, training scores kept rising while actual code test performance got worse and worse. The pitfall of Reward Hacking is explained with a single curve. Some commenters under the repost mentioned they had fallen into the same trap before, right at the point of advancing to more capable Agents.

The Resources section is fairly comprehensive. Each step has a YouTube livestream, lecture slides, and TRL documentation links; steps 2 through 4 also come with ready-to-use training scripts, datasets, and a Trackio dashboard. The code for step 1 is in burtenshaw's gemma4-pi-mono repository. The course homepage also has a 20-page overview deck and a glossary explaining terms like harness and scaffold.

Someone asked what level of GPU is needed to run the whole course, and the author gave a very practical answer: it depends on the scale of the model you want to train. This roadmap doesn't promise instant results; what it does is lay out the boundaries and pitfalls of each stage. People who already know how to build Agents and want to move toward Agent RL and training environments can follow it step by step.

Course URL: https://huggingface.co/blog/huggingface/train-agents