11 Open-Source Models Read 300 Doctors' Handwritten Drug Names: Best Got 76 Right, and They Invent English Words When They Misread

300 real handwritten drug names, 11 open-source models, thinking disabled, counting only exact matches. The top score was Gemma 4 31B, which read 76 of the 300 perfectly. A 0.8B OCR model got 65 right on a single NVIDIA GPU. Maziyar Panahi posted the test on X; the dataset came from RxHandBD and Mendeley Data and is licensed CC BY 4.0.

Of the 300 samples, only 115 were read correctly by at least one model. Some of the dataset labels themselves are wrong, so no model could reach 300. The chart lists the top 7 of the 11 open-source models: Gemma 4 31B 76, Xiaomi-OCR-O 65, OvisOCR3 61, MiMo-V2.6-Flash 60, MiniCPM-V 4.7 52, Nemotron 3.5 Super VL 49, Moondream 3.1 47. The test conditions are stated clearly: exact match, thinking off, hosted runs pinned to a single provider, no fallback, and GPUs running on Hugging Face Jobs.

The really troubling part is how they misread. Maziyar Panahi says that when a model cannot read a drug name, it often writes an English word. Nutridex becomes Nut index, Calbon becomes Cunning, and pregmo becomes Physiology. Every model showed this behavior, between 9 and 88 times out of 300. These three examples gave the same results in two independent runs. Gemma 4 wrote illegible 18 times and did not force a guess. The original post says this is the instinct you want in a prescription setting.

One commenter focused on the 9 to 88 substitutions: writing illegible is safe; turning pregmo into Physiology is not. They asked whether the model with the fewest substitutions also scored high on exact match. The original post did not answer.

The accuracy comparison is also worth a look. Reading a doctor's notes from the Civil War era in 1863 letter by letter reaches 98%. Reading today's prescriptions reaches only 71%. Maziyar Panahi's explanation is that the 1863 notes have sentences to rely on, while isolated drug names have no context. High-quality, doctor-verified handwriting samples are also scarce. Radiologist Dr Datta M.D. said in a reply that frontier models perform well, but local models are not there yet.

This test does not prove that AI can replace pharmacists. It is more like a failure-mode check. In a prescription setting, a model saying 'illegible' is more useful than fluently inventing a drug name. The gap between open-source models is not only in the top score, but also in whether they are willing to admit they do not know. Pharmacists are safe for now, but people building medical OCR and agents should first look at what a model writes into a prescription when it misreads.