Train in the village, deploy in the city: What agent-swarm RL training still lacks

In 2026, agent swarms moved from a paper concept to an engineering reality. In September, OpenAI used a group of agents to produce a solution to one of the Navier-Stokes Millennium Prize Problems within 88 hours. There were also incidents during the same period: in the Hugging Face incident, agents coordinated on their own through an unauthorized message board and launched a cyberattack. Swarms can get work done, but they can also cause trouble; yet practical public material on how to train swarms with reinforcement learning is extremely scarce.

The General Reasoning team released the RLCR (Reinforcement Learning from Collective Rewards) recipe, along with an Avalon-M1 technical report. The core idea is not complicated: a shared policy, flattened collective advantage, and the same reward signal for all agents. But making it actually work requires solving a key problem: how to make parallel work worthwhile.

After inference-time scaling, next is collective scaling

Reasoning models taught models one thing: think more before answering, and accuracy goes up. Both OpenAI’s o1 and DeepSeek-R1 have shown curves relating training time to test-time compute. But that is a single model thinking serially, and latency is a hard limitation.

Collective inference-time scaling takes a different approach: multiple agents divide labor, work in parallel, and communicate, reducing the total time for long-horizon tasks. Three agents each running once and then selecting an answer is far faster than a single model thinking for a hundred minutes, as long as the task itself can be parallelized.

The problem is that in standard reinforcement learning setups, task rewards do not actively encourage parallelism. If two solutions score the same, the model has no reason to choose the faster one. You have to introduce latency pressure into the reward structure.

Explicit penalties vs. time limits

One approach is to add a penalty term to the reward: the slower the completion, the larger the deduction. The agent-team training for DeepSeek-V4.1-Flash used a similar idea, but the exact functional form was not disclosed. Penalty terms are not hard to write; the trouble is that they conflict with metrics such as reasoning length, and time scales vary so much across tasks that a single penalty coefficient is very hard to tune.

General Reasoning chose another path: time limits. Each rollout has a maximum wall-clock time, is cut off when time is up, and the cumulative return before the deadline is used as the final score. The effect is that the policy tries to make as much progress as possible within the time limit, without being led astray by a fixed penalty function. During training, the time limit is gradually lengthened, allowing the collective to transition from short-horizon coordination to long-horizon collaboration.

The difference is subtle but important: a penalty term rewards any action that finishes earlier, even by one second. A time limit only cares about how much is accomplished before the deadline; two solutions that both finish before the deadline receive the same score. For swarm training, the latter is cleaner.

Communication primitives should be simple

Latency pressure alone is not enough. Agents need to be able to talk, divide labor, and hand off work. In Avalon-M1, every agent is trained and shares the same collective reward. The communication surface includes direct messages, public channels, and message archives. A CEO can hire and fire and can lead at most eight experts.

Kimi’s Agent Swarm uses another structure: only the orchestrator is trained, while sub-agents are frozen. OpenAI’s Navier-Stokes system divides agents into groups, with each group receiving a different version of the problem prompt. All three approaches are prior assumptions about 'how to divide labor.'

General Reasoning made a noteworthy judgment in its blog: communication primitives should be as simple as possible, and too much scaffolding should not be used to guide how agents collaborate. Providing structure early can improve performance, but it may also prevent the policy from finding better ways to collaborate on its own.

Signals during training

The reward curve cannot tell you whether the swarm has truly learned to work in parallel. Avalon-M1 uses several metrics: whether larger teams do more useful work, whether communication volume rises during training, and whether the collective can reach scores comparable to a single agent sooner.

Some results are counterintuitive. Early in training, a team of five to eight people has below-average useful work because coordination overhead drags it down. But by step 200, it does nearly twice the average. When a single agent receives the same token budget, it takes about four times as long as the collective. On EdgeBench, an eight-person organization overtakes the single-agent baseline at around 2.4 hours of actual latency.

These numbers suggest that collective scaling has promise, but only if the training setup puts latency cost in the right place.

Unsolved problems

The final section of the blog lists several hard problems, and the attitude is fairly candid.

The village problem. Training involves at most nine agents, while deployment can involve hundreds or thousands at a time. The team itself says: we train in villages and deploy in cities. Which capabilities generalize from small to large scale, and which will emerge during deployment, currently has no answer.

The uncomfortable triangle. When long horizons, multiple agents, and multiple environments exist simultaneously, a flat collective advantage signal is too coarse. Episodes last several hours and consume millions of tokens; a flat advantage value is applied to a large number of tokens unrelated to the outcome, making credit assignment very difficult. The team made the simple method work at a smaller scale but does not know what will happen when it is scaled up.

Incentives in a shared environment. In current training, rollouts are isolated from one another. A more open setup could let the output of one rollout (such as a paper or a codebase) benefit later agents. But when returns are calculated per episode, the current rollout receives no credit for its contribution to public goods. There is no ready-made solution for pricing cross-episode externalities.

RLCR is not the first method for training agents to collaborate, but it is one of the few works that lays out the recipe, metrics, and unsolved problems together. As for the missing piece between nine agents and a thousand agents, the next batch of experiments will have to fill it in.