Can't Write Reward Functions Anymore? Let LLM Judges Do the Scoring: RULER Pushes GRPO to 2048

Karpathy previously judged that a single reward number tells an agent too little. In complex tasks, one successful outcome can hide wasted steps and bad tool calls. That judgment is now being engineered into practice.

Traditional RLHF trains a reward model from human preferences, and PPO has to maintain the policy, reference policy, reward model, and critic at the same time. DeepSeek R1 used RLVR and GRPO to make verifiable tasks work: math problems are checked against answers, code is run against tests, and the reward is 0 or 1. GRPO normalizes multiple responses to the same prompt within a group, eliminating the critic.

The problem shows up in agent tasks. RAG, customer service, summarization, and tool calling are hard to score with string matching or a compiler. Hand-written reward functions can work, but the rules have to anticipate behavior in advance, weights are hard to tune, and the whole thing must be rewritten when the workflow changes. More troublesome: when training produces bad behavior, it is hard to tell whether the reward function measures the wrong thing or whether the problem lies in the data or hyperparameters.

In a Daily Dose of Data Science article, Avi Chawla introduced RULER in OpenPipe ART. It writes scoring criteria in natural language and has an LLM judge multiple trajectories. It generates 4 to 8 trajectories for the same scenario; the judge model reads the system prompt, understands what the agent should do, and then scores those trajectories relatively. ART fills the scores into Trajectory and hands them to GRPO for training.

The key is relative scoring. LLMs tend to drift when doing absolute scoring; comparing “which of these 4 better follows the system prompt” is more stable. GRPO already normalizes rewards within a group, and it cares about which one is above average and which is below average. RULER's relative rankings line up exactly with this training signal.

They used this setup to train Qwen3 1.4B to play 2048. The agent looks at the board and chooses a direction each step. They did not write a set of rules to score moves. Instead, they described what the judge should look for and let RULER compare complete attempts. The final reward is still numeric; what changes is how the reward is defined. If they find the judge is missing some behavior, they change the description and then check whether its judgments match their intent.

In the comments, someone asked how often the judge model and the game's raw score disagree on the same batch of runs. That question should be checked first. LLM judges also hallucinate, and their preferences can be gamed by the policy. RULER provides debug output, disk caching, and custom rubrics. You can start with a cheap judge model—for example, Qwen3 32B is enough for many tasks and costs less than o3. Whether the judging criteria are reliable still needs spot checks.

Anthropic's Constitutional AI uses written principles to let AI evaluate outputs, and OpenAI's Universal Verifiers is also moving toward general judge signals, though details are not public. The direction is clear: optimization algorithms are already good enough; defining the reward signal has become the new bottleneck.

The source of reward signals is migrating from hand-written code to readable judging instructions. This migration does not eliminate subjectivity; it puts subjectivity into text, where it is easier to modify and inspect. For agent training, the next hard problem is defining what counts as good and how to verify that the judge is actually scoring according to that definition.

GitHub: https://github.com/OpenPipe/ART

Article: http://x.com/i/article/2048280670825496576