Hugging Face Breaks Agent Post-Training into 5 Steps, with Code and Livestreams from SFT to RL

Hugging Face's merve shared a post by Sergio Paniego about how to do post-training for agents. They broke what looks like a complex process into 5 steps, each with a flow diagram, livestream class, code, and documentation.

First, some background. Post-training refers to the training stage after pretraining, including supervised fine-tuning, distillation, and reinforcement learning. These stages turn a pretrained model into a model that can follow instructions, use tools, and solve specific tasks. When a model works as an agent, it runs many rounds, calls tools, reads results, and decides the next step. Agent post-training must take the entire multi-turn sequence into account, not just focus on a single response. Hugging Face's course goes in order, with each stage solving the problems left by the previous one.

1. SFT: First, teach the model to copy the correct answers

The first step is supervised fine-tuning. The model learns to replicate agent trajectories. A trajectory is the complete record of a stronger model solving a task, including every tool call and returned result. This method is cheap and stable, but the model only sees correct trajectories and never sees its own attempts. If it makes a mistake while solving a problem, it has not learned how to recover from the error. Online distillation and RL later fill this gap.

The corresponding trainer is SFTTrainer; prepare a dataset of agent trajectories and you can start.

2. Distillation: Let the teacher model score every token

The second step is distillation. The teacher model supervises the student model in two ways: the teacher writes data for the student to learn from, or it gives a probability distribution for the tokens generated by the student. The teacher is usually a larger model, or several expert models can be distilled into one student (MOPD), or the same model can be used with extra context for self-distillation.

The distillation signal is dense, one value per token, so training requires fewer samples than RL. The cost is that the student is limited by the teacher's knowledge scope, and online distillation still has to run the teacher at every training step. RL replaces the teacher with a reward function, checks only the final result, and saves the cost of teacher inference.

The trainer is DistillationTrainer, which needs prompts and a teacher model.

3. RL: Use scores instead of a teacher

The third step is reinforcement learning. The model learns from scores on its own outputs: it writes an answer, a reward function scores it, and training makes high-scoring answers more likely. This does not require a teacher, only the ability to check answers.

This step uses the GRPO algorithm. It samples multiple answers for each prompt, scores them all, and increases the probability of answers above the group average. Comparing within the group tells the model which attempts are better, without training a separate value model to estimate the baseline.

The RL signal is weaker than distillation: one number for the whole answer, not one per token. The model needs many attempts to learn. With an ordinary reward function, each answer is single-turn and scored only once. An environment lets the model act several times and see the result of each action.

The trainer is GRPOTrainer, which needs prompts and a reward function.

4. RL environments: Let the model act over multiple turns

The fourth step changes the source of reward from a function to an environment. The environment keeps state, exposes tools, and responds across multiple turns. It runs tools, records what changed at each step, and finally checks the result, for example by running the test cases for a coding task.

Each episode is much longer than a single answer, so generating rollouts becomes the slowest part of training. This step requires tasks and an OpenEnv environment, and the trainer is still GRPOTrainer.

5. Harness and scale: Train in the real runtime environment

In the final step, each rollout is run end-to-end through the existing agent harness in a remote sandbox. By training with the same harness used in actual deployment, the model learns in the setting where it will run in the future. Generation and training run on different GPUs, so long episodes do not leave the trainer waiting idle. The trainer sends the updated weights back to vLLM.

The trainer switches to AsyncGRPOTrainer, which needs tasks, an environment, and an agent harness.

What is the value of this course?

The technology itself is not new; SFT, distillation, GRPO, and environment training are all existing methods. The value lies in the ordering and the explanation of boundaries: when SFT is enough, when environments and harnesses are necessary, and what the prerequisites and limitations of each step are, all written clearly. Each step comes with a livestream, code, and documentation, so people who want to get hands-on can follow along directly.

The course homepage is here, and the playlist of four livestreams is here.