No 8-GPU Setup? You Can Still Run Agentic RL—This Repo Connects Paper Formulas to Training Code

People who want to learn Agentic RL often get stuck on environments, GPU count, and coupling with training frameworks; the formulas are usually not the first obstacle. Agentic-RL Lab, built by Datawhale member Song Zhixue, has a very direct goal: break apart LLM and Agent reinforcement learning algorithms and provide runnable code.

Agentic RL can be understood as teaching a model to search, call code tools, make decisions in an environment, and then train it with reinforcement learning across multi-turn tasks. It sounds easy, but once you actually start, questions like how to prepare data, how to assign rewards, what the loss is optimizing, and how an Agent trajectory participates in training all become concrete problems.

The repo covers a set of algorithms and experiments including GRPO, DAPO, GSPO, Search-R1, ReTool, AgentOPSD, ALFWorld, TEMPO, and Spec-o3. The project uses PyTRIO's Train API to lower the training engineering barrier and SwanLab for experiment tracking. Paper sources, core variables, data processing, training loops, and logging are all kept together. When you read about an algorithm, there is runnable code right next to it.

For example, the GRPO chapter compares three losses—importance_sampling, ppo, and cispo—on GSM8K. Search-R1 trains multi-turn search, ReTool trains a model to call code tools, ALFWorld involves doing household chores, and Spec-o3 uses spectral data to find stars. If you want to study a reward design or a loss change, you can first get a corresponding experiment running and then modify it from there.

The project author mentions in the repo that he was initially discouraged by the complexity of 4-GPU and 8-GPU setups and integrated training/inference, but after encountering Tinker and PyTrio's Train API, it took him less than two months to reproduce the algorithms from these 14 papers. This point is worth noting: once training infrastructure strips away engineering noise, there is much less core content for learners to focus on.

In the comments, someone asked whether GSM8K is too simple. The author replied that GSM8K is too simple and generally requires a harder task. Someone else asked whether the reward functions used by these algorithms were tuned separately or shared a common baseline. The author replied that tuning them separately works better. Another reader wanted reference logs after each experiment was successfully run, and the author suggested filing an issue requesting logs. These details are more useful than just listing algorithm names.

The Spec-o3 section provides a set of results: on the same batch of 256 questions, Qwen3.5-4B answered 175 correctly, for 68.36% accuracy, while Qwen3.5-9B answered 191 correctly, for 74.61% accuracy; Macro F1 was 70.93% and 75.94%, respectively. The full comparison and evaluation setup are in the Spec-o3 Blog.

Quick start is not complicated:

git clone https://github.com/datawhalechina/agentic-rl-lab.git
cd agentic-rl-lab
uv sync

When running the ALFWorld, TEMPO, or AgentOPSD chapters, you need to additionally install the TextWorld environment dependencies:

uv sync --extra alfworld

Search-R1 directly calls DeepSeek's official Search API using httpx, so no additional search wrapper needs to be installed. Basic dependencies are managed with uv. If you want to pull out an individual demo script and run it in your own project, you can also install according to the repo instructions.

The repo also mentions a user story: someone listed the repo's AgentOPSD chapter on their interview résumé and later joined MemTensor. This detail is interesting, but it doesn't need to be overinterpreted. The more practical value is that it moves learning materials from just reading formulas to being able to run experiments.

There is a Chinese experiment manual and an English version available online. The project is at Datawhale Agentic-RL Lab. If you want to learn Agentic RL, getting one chapter you're interested in running is more effective than continuing to bookmark papers.