12 open-source models chose induction of labor 36 times, but the answer key says cesarean section—who is wrong?

On a MedQA question about a 32-week pregnant woman with preterm premature rupture of membranes and contractions, the answer key is cesarean section. Maziyar Panahi sent this question to 12 open-source models, running each question 3 times. Across 36 responses, all chose induction of labor.

The question asks for the most appropriate next step. Among the options, C is to give betamethasone, ampicillin, and perform cesarean section; D is to give betamethasone, ampicillin, and perform induction of labor. The answer key chooses C, while the models all choose D.

This is not a bizarre question picked at random. In a reply, Panahi added that this is question 1230 out of 1273 in the MedQA test set. The test involved 12 open-source models from 10 labs, including Kimi, Qwen, DeepSeek, GLM, Mistral, Gemma, and others. Same prompt, temperature 0, each model run 3 times.

The results can be viewed on two levels.

First, the models' overall performance is not bad. The 12 models scored between 92.8% and 96.4% on the full exam. Kimi K3 led. Qwen3.8 27B is small enough in parameter size to fit on a single GPU and still scored 94.5%. There were 1013 questions that all 12 models answered correctly in every run.

Second, the models were wrong in a very orderly way. Among the 1273 questions, there are 6 questions where all 12 models completely agreed with one another, but the answer key chose another option. This obstetrics question is one of them. The models showed no disagreement and no random fluctuation; all 36 runs pointed to induction of labor.

What is most worth examining here is not how many points the models scored again, but the shape of the errors behind high scores. High exam scores show that the models have seen many medical questions and guideline statements. But when a question lands in a specific clinical situation—especially one such as 32-week preterm premature rupture of membranes, existing contractions, and good fetal heart rate reactivity, where maternal and fetal risks must be weighed—the models tend to favor a certain path, and they favor it unusually consistently.

The answer key is not necessarily inherently correct. MedQA's answer key is a dataset annotation, and medical questions often involve differences in guideline versions, institutional protocols, and clinical judgment. Panahi directly asked doctors who is right, which shows that the question itself has room for discussion. The models all choosing induction of labor may be wrong, or they may have landed on another approach to clinical management.

But unanimous agreement among models likewise cannot be taken as proof of correctness. The 12 models come from different labs, with different training data, architectures, and post-training strategies, yet they give the same answer on the same question. This looks more like an evaluation signal: in certain medical situations, model consistency may come from similar corpus distributions rather than independent reasoning.

For developers, this is more useful than simply looking at leaderboard rankings. A model scoring 94% on MedQA does not mean it is reliable on controversial questions, borderline questions, or questions requiring clinical trade-offs. For ordinary users, this also serves as a reminder: models answering questions is like taking an exam, but clinical decisions must face actual people.

Panahi's original post is here: X post. If doctors see this question, they are welcome to correct the answer key and also take a look at why the models are so uniform.