2B Model Goes from 5.5% to 34.4%—Are 500 Agent Trajectories Enough? First, Look at How the Data Is Split

Hugging Face TRL has released a post-training example: let Qwen3.5-397B-A17B act as the teacher, run data analysis tasks in OpenCode, keep the correct trajectories, and then perform SFT on Qwen3.5-2B.

Here are the results first: on held-out tasks, the 2B model goes from 5.5% to 34.4%, while the teacher is 74.2%. The process video is below.

The tasks come from SmolDataEnvs. They are data analysis questions on real Kaggle datasets, such as “What proportion of matches end in a draw?” Each question is a Harbor task: a question, dataset files, a standard answer, and a deterministic grader that scores the answer.txt written by the agent as 1 or 0. The training set has 5000 questions, and eval has 144 questions.

The agent is OpenCode, an open-source coding agent with file tools such as bash and read. OpenEnv runs it inside Hugging Face Sandbox to execute tasks and records every model call: the messages sent, the tools provided, and the model's responses. The training data is these records.

The whole pipeline has 5 steps:

1. Prepare tasks.
2. The teacher model runs the training tasks and keeps every rollout.
3. Keep only the rollouts that were done correctly to build the SFT dataset.
4. Fine-tune the student model with TRL's SFTTrainer.
5. Serve the student model with vLLM and score it on eval/test.

The teacher model goes through Inference Providers; the job that generates trajectories does not need a GPU. It solves 78% of the clean training tasks, with a median of 6 turns and about 2 minutes per question. Each trajectory is about 10k tokens, most of which are OpenCode's system prompt and tool schemas; only 4% to 8% actually participates in training. With about 500 trajectories, training for 3 epochs on a single H200 takes about 16 minutes. assistant_only_loss=True puts the loss on assistant turns, so the student learns the teacher's tool calls and answers, not the tool outputs. Trajectories longer than 16k tokens are dropped directly without truncation, because truncation would cut off the final answer.

The key detail is the data split. The original article says train/eval/test are sampled by task, not by Kaggle dataset. Of the 144 eval tasks, 122 use a Kaggle dataset that also appeared in some training task; of the 250 test tasks, 185 do the same. The student may have already seen these files, column names, and structures.

A commenter, Sagar Tanur, followed up in the replies: how are the held-out tasks split? If they come from the same batch of repos as the solved trajectories, part of the gain may be the model learning the repo layout rather than the agent getting better. Deep also reminded readers to check whether the held-out tasks come from different repos.

This example handles it by adding an --exclude-datasets-of filter, skipping training tasks that share a Kaggle dataset with the eval split; 2879 of the 5000 questions remain. The final evaluation uses only 307 tasks whose Kaggle datasets were not used by any training trajectory, including 144 eval tasks and 163 test tasks.

What happens without this filter? The original article gives a comparison: the student model scores 17.3% on eval tasks without shared datasets and 42.4% on tasks with shared datasets. The gap is straightforward: the model first recognizes the files, then talks about solving the problem.

The number of trajectories also has diminishing returns. With 473 trajectories, eval is 33.3% and test is 34.4%; with 963, 32.6% and 38.7%; with 2238, 36.8% and 38.7%. Most of the gain comes from the first few hundred; adding several times more data later only adds 3 to 4 points. On 144 and 163 tasks, differences of a few points are within noise.

This SFT is positioned as the starting point for RL. The 2B model can score on fewer than one in ten questions, and on two-thirds of tasks it does not write an answer at all. The teacher first teaches it to use tools and when to answer; then RL starts from a model that can answer most tasks and solve one-third of them. The SFT here is sequence-level distillation: the student only imitates the teacher's text and does not get the teacher's logits. TRL's on-policy distillation trainers, such as AsyncDistillationTrainer, train on the student's own samples using the teacher's per-token distribution and require serving the teacher alongside the student.

To reproduce directly, you can skip to step 4 and use the released trajectories: --dataset sergiopaniego/smoldataenvs-opencode-traces --dataset-config 500, or choose 1k or full. The complete scripts and commands are in the Hugging Face TRL examples: https://github.com/huggingface/trl/tree/main/examples/sft_agent_traces .