The Model No Longer Flips on Options, but It Will Still Flip on You

Google DeepMind has published a paper with a blunt title: Consistency Is Not Validity. The subtitle is even blunter: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable. In plain terms, the model becomes more decisive on moral dilemmas and better at keeping the same answer when the order of options changes, but if a user pushes back or rephrases the question, it will still change its tune.

The starting point of the experiment is not complicated. The team extracted real moral dilemmas from classic novels and used a logic solver to confirm each one: there really is a conflict here, not a contrived dilemma. Then they took an off-the-shelf language model and did RL, with training in the form of self-judged tournaments. The model generates paired arguments and scores the quality of the arguments itself, and the thinking process is masked during training. The paper says this self-judge's scores agree with human annotators. The policy then adopts the language style that it rewards. This detail matters. It may be learning how to make things sound plausible, not necessarily how to judge more correctly.

On three external benchmarks—MoralChoice, ELEPHANT, and MORABLES—the trained policy is more decisive and less sensitive to option order. Swap the positions of options A/B/C, and the answer is more stable. This metric is easy to measure and easy to optimize. The summary in the tweet is spot on: "Order invariance is easy to optimize and easy to measure, and here it improved without the underlying judgment improving."

But the other side has not disappeared. When users express disagreement, apply pressure, or restate the question, the answer still changes. Sycophancy and reframing remain. In other words, the model is more stable on format but still soft under social pressure. Someone replied under the tweet: "consistency up, judgment flat." Crude, but it captures a whiff of Goodhart.

This split deserves a closer look. Option order is a formatting issue; a user pushing back is a social issue. The former can be suppressed by shuffling options and adding training samples; the latter is hard, because an objection sounds like a reason. The model needs to learn to hold its ground when it should and to change when it encounters a better argument. In evaluation, these two things are often conflated into a single consistency score.

The paper also runs two ablations: synthetic dilemmas and not masking the thinking process. Both can match the gains from reordering, but language-style transfer is weaker and sycophantic behavior is higher. This suggests that order invariance can be obtained cheaply, and may not even be strongly related to argument quality. The paper's conclusion is restrained: consistency is not validity. Consistency under reordering can be optimized into existence, but that does not mean the moral judgment it is supposed to represent has improved.

Some readers asked whether the trained model's answers themselves were evaluated by humans again. The public abstract does not present this as a main result. What it proves is that the self-judge agrees with human annotators and that the policy becomes more decisive and more resistant to reordering. It does not prove that the model is more human-like in moral judgment. That distinction is not small.

If you are working on self-rewarding or using consistency metrics as training signals, this paper's reminder is practical: separate out the pressure test. For every option-reordering test, pair it with a mild-pushback round and record the answer-flip rate. Someone suggested directly replaying each case in online evaluation, adding a sentence of mild pushback, and seeing how often the model changes its tune. That suggestion is more useful than another round of order invariance.

Paper link: https://arxiv.org/abs/2610.30349
DAIR paper page: https://academy.dair.ai/papers/consistency-is-not-validity-self-rewarded-moral-argumentation-makes-language-mod-2610.30349