The Secret Algorithm of RL Data Companies? Commenters Call It a Perpetual Motion Machine

Michael R. Bock posted a flowchart on X, saying it looks too simple to be true, but it is the secret of every RL environment and data company today, and also the algorithm behind RSI.

Let’s fill in the context first. An RL environment can be understood as a training ground prepared for reinforcement learning models; data companies are responsible for creating tasks, validating tasks, scoring models, and feeding the results back. RSI is recursive self-improvement, in which models participate in improving the next generation of models or the training process. The flowchart is not complicated: a generator model creates new tasks, a validator checks whether the tasks are correct, an improver optimizes the generator prompt based on feedback, the model under test runs on the tasks, a scorer assigns scores, and frontier tasks are placed to one side as a difficulty boundary. Once looped, it is a pipeline of generation, validation, scoring, and further improvement.

Problems quickly appeared in the comments. One commenter said the method is naive and looks very expensive. Another directly called it a perpetual motion machine. Someone else pointed out that the diagram is missing a stressed human in the loop, and the author replied “definitely” with a laughing-crying emoji. That reaction is more informative than the flowchart itself. Every box in the diagram can be drawn; the real difficulty is who writes the validator, how the scorer is calibrated, and how many holes humans have to patch once tasks become harder.

There was also debate about model capability. One commenter asked: if the models that the generator or validator relies on are not good enough, how can you generate sufficiently good RL environments or validation tasks? They mentioned models such as Fable 5. Michael’s reply was that Opus 5.5 is good. This detail shows that the debate is not only about the process, but also about whether the underlying models can support validation quality.

A sharper question came from auditing. One commenter asked: once the generator surpasses the validator, who audits the validator? The flowchart has no answer. Once the validator becomes the bottleneck, the loop degenerates into the generator making the questions and grading the answers itself. Costs also balloon here, because validation and human review are often more expensive than generation.

There was positive feedback as well. One commenter said they had already implemented about 90% of the process and were planning to improve it further. In another reply, the author asked “open source?”. This roughly represents two types of readers: one type thinks the diagram is too idealized, while the other has already run similar structures in engineering and knows which box consumes the most human effort.

What really holds this diagram back may not be the name of the algorithm, but the quality and budget of the validator. A flowchart can be drawn very cleanly; validators and review manpower cannot be open-sourced.