Multiple LLM judges scoring: simple averaging is actually worse

Using LLMs as judges to score open-ended tasks, then using these scores for reinforcement learning, is a method called Rubrics as Rewards, or RaR for short. In the paper Judging the Judges: Confidence and Disagreement in Rubrics as Rewards, UC Berkeley's Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell, and Yutong Bai ask a different question: if a single judge is replaced by multiple judges, will the reward become more stable? The conclusion is somewhat counterintuitive: directly averaging multiple judges' scores does not beat the best single judge; to make multiple judges useful, confidence and disagreement must be handled.

First, some background. RaR targets tasks that are hard to verify with standard answers: first provide a structured rubric, then have an LLM judge score the model output according to the rubric, and this score enters RL training. Prior work mostly changed the rubric or made a single judge stronger. This paper shifts attention to judge identity and judge combinations.

The paper trains Qwen3-4B-Base on RaR-Medicine and RaR-Science using GRPO, swaps in five frontier models and open-weight models as judges, and finally evaluates on HealthBench and GPQA Diamond. Single-judge identity leads to considerable score differences: on HealthBench, single judges range from 0.296 to 0.339; on GPQA Diamond, from 0.345 to 0.406. On HealthBench, Qwen3-4B-Base without RL is 0.257, and the tuned reference model is 0.308.

The paper proposes RaRE, short for Rubrics as Rewards with Ensembles. The two mechanisms can be used separately or combined.

The first is soft-Likert reward. Each judge scores on a 1-to-5 scale. The method reads the top-5 logprobs at the position of the rating number, renormalizes them, and obtains the expected score E_j and confidence c_j=1/(1+Var_j). The reward is a confidence-weighted average: r_soft = (1/L) * Σ c_j E_j / Σ c_j, L=5. More confident judges get larger weights; judges for which logprobs are unavailable fall back to integer scores.

The second is drop-on-disagreement. For each rollout, compute the range of the judges' normalized scores d=(max_j r_j - min_j r_j)/(L-1). When d exceeds threshold τ, this rollout's reward is replaced by the mean of unmasked samples in the same group, the advantage in GRPO becomes 0, and it contributes no gradient. On the HealthBench 1K subset, the Gemini+DeepSeek group drops about 5% at τ=0.30, with score 0.330; the paper's best RaRE uses 0.30 for medicine and 0.40 for science.

Why doesn't direct averaging work? The paper gives numbers. On HealthBench, direct averaging for Gemini+DeepSeek is 0.307, below the best single judge, GPT-4o-mini, at 0.339. On GPQA Diamond, Gemini+DeepSeek averaging is 0.363, also below the best single judge at 0.406. When scores are averaged, low-confidence and high-disagreement judgments are mixed in as well, diluting the reward signal.

RaRE combines them differently. The best configuration is DeepSeek-V3.1 + GPT-4o-mini with soft-Likert plus drop. On HealthBench it is 0.364 vs. the best single judge at 0.339; on GPQA Diamond it is 0.412 vs. the best single judge at 0.406. It beats all single-judge RaR variants on both benchmarks.

Breaking it down by HealthBench topic, RaRE does not average away the judges' strengths; it inherits them. DeepSeek-V3.1 + GPT-4o-mini achieves the highest gain in the row on all seven topics: Comm., Complex, Context, Emerg., Global, Data, and Hedging. For example, Global +0.131, Hedging +0.134, Context +0.118.

The ablations also illustrate the point. Changing confidence aggregation from inverse-variance weighted averaging to top-logit weighted averaging drops the score from 0.336 to 0.214; changing to top-logit winner-takes-all drops it to 0.198, a difference of 12 to 14 percentage points. The threshold τ is not better when stricter either: at τ=0.15, 14% are dropped and the score falls to 0.251; at τ=0.30, 5% are dropped and the score is 0.330.

In the comments, someone asked which HealthBench topics cause the most disagreement, but the tweet thread did not elaborate. Others mentioned that the key is to account for disagreement and confidence; otherwise, adding more judges may just average the noise. Someone also asked whether there is a separate ablation of the contributions of judge diversity and confidence weighting. The paper shows comparisons of single judges, two-judge pairings, and confidence aggregation methods; finer diversity ablations require looking through the paper and code.

The code repository is very specific: training uses Slurm scripts, a single node with 8 80GB GPUs, and evaluation uses a single GPU; the shared recipe is 1000 training prompts, 8 rollouts per prompt, 100 steps, learning rate 5e-6, full fine-tuning, max completion 1024 tokens, and vLLM colocate. Soft-Likert requires the judge to return logprobs; these were available when DeepSeek-V3.1 was provided by Together, but this endpoint no longer supports them.

Paper: https://arxiv.org/abs/2610.21179

Code: https://github.com/michael-tyl/RaRE

Project page: https://anneharrington.github.io/judging-the-judges/