Training agents: don't rush into RL; Hugging Face breaks post-training into 5 steps

Sergio Paniego asked on X how to post-train an agent. Hugging Face turned the answer into a five-step course, from SFT to RL and then to a real harness, with diagrams, livestreams, code, and docs for each step. The course homepage is here.

Post-training refers to the training stage after pretraining, turning a base model into one that follows instructions, uses tools, and can complete specific tasks. An agent calls tools over multiple turns, reads results, and decides the next step; training must cover the whole process, not just a single response.

This diagram puts four learning signals together: SFT, distillation, RL, and RL environments. What signal the model learns from determines what you need to prepare later. The five steps correspond to SFTTrainer, DistillationTrainer, GRPOTrainer, GRPOTrainer, and AsyncGRPOTrainer.

1. SFT

The model learns to imitate agent traces produced by a stronger model—complete task records including every tool call and result. It is cheap and stable. The problem is that the model only sees correct trajectories, not its own failures, so it doesn't know how to recover after making a mistake. This corresponds to TRL's SFTTrainer and requires a dataset of agent traces.

Watch the livestream: Training Agents 1; the code is in the livestream description. Docs: SFT Trainer.

2. Distillation

The teacher model supervises the student by writing data or by giving probabilities on tokens generated by the student itself. The teacher may be a larger model, multiple expert models distilled into one student, or the same model with extra context for self-distillation. The signal is dense—one value per token—and requires fewer samples than RL. The cost is that the student is capped by the teacher's knowledge; on-policy distillation must run the teacher at every step. RL replaces the teacher with a reward and only checks the result.

Watch the livestream: Training Agents 2. Read: A Brief History of Distillation, Distillation in 2026. Docs: Distillation Trainer.

3. RL

The model learns from scores on its own outputs. It writes an answer, a reward function scores it, and high-scoring answers become more likely. This step needs no teacher—only a way to check answers. The course uses GRPO: sample multiple answers for the same prompt, score them all, and increase the probability of answers above the group average. In-group comparison tells the model which attempt was better without training a separate value model. The signal is weaker than distillation—one score per answer—and requires many attempts. Under a standard reward function, each answer is single-turn and scored only once.

Watch the livestream: Training Agents 3. Read: How frontier models train on outcomes in 2026. Docs: GRPO Trainer.

4. RL Environments

Reward comes from acting in an environment. The environment keeps state, exposes tools, supports multi-turn responses, runs tools, records changes, and checks the final result—for example, by running code tests. Each episode is longer than a single answer, so generating rollouts becomes the slowest part of training.

Watch the livestream: Training Agents 4. Read: Why RL environments work now. Docs: TRL + OpenEnv, OpenEnv.

5. Harness and Scale

Each rollout is a complete agent run, going through the existing agent harness and running in a remote sandbox. Training and real use share the same harness, so the model learns in the setting where it will later run. Generation and training run on different GPUs; long episodes don't leave the trainer idle, and updated weights are sent back to vLLM. This corresponds to AsyncGRPOTrainer.

View the slides: Training a coding agent through a harness you did not write. Read: Multi-harness RL guide, One sandbox per rollout. Docs: Async GRPO Trainer, OpenEnv harnesses.

The subtext of this order is: first learn reproducible trajectories, then compress teacher signals, then explore with rewards, and finally bring the training environment close to the actual runtime environment. Jumping straight to RL easily overlooks the agent's multi-turn structure. SFT is not the endpoint, but it is the starting point for the later steps. The truly hard part is the environment and the harness, not the name of a particular trainer.