Writing Its Own Test and Taking It: This Company's OCR Model Took First Place

OmniParseBench is now a native benchmark on Papers with Code. Niels Rogge announced it in a tweet and released the full leaderboard.

The benchmark was developed by Datalab. It contains 16,288 yes/no tests covering 2,937 document pages, 2,343 files, and 92 languages. Unlike the commonly used olmOCR-bench, it scores tables, mathematical formulas, handwriting, rotated pages, forms, small text, and degraded scans separately, and each test can pinpoint exactly where on the page an error occurred.

At the same time, Datalab founder Vik Paruchuri posted another tweet: they ran 17 open-source OCR models on the same tests, deploying each model with its vendor's recommended configuration.

Ranking results:

chandra-ocr-2 scored 90.1, higher than Mistral OCR's 89.5, Claude Sonnet 5.5's 88.7, and GPT-5.6 Sol's 87.8. It is only 3.4 points away from the top score of 93.5 from Datalab's own API.

Here's the problem. chandra-ocr-2 is Datalab's own model. So is Surya OCR 2. Two of the top three on the leaderboard come from the evaluator itself.

Vik himself was the first to say in a reply: "When your own model is at the top of the leaderboard, it's always a bit suspicious." He then released all adapters and the dataset; each model's deployment parameters, prompts, and output parsing methods are listed in the blog, and the scores can be audited item by item on GitHub and HuggingFace.

Another set of data worth looking at is tables. Many people assume that OCR models fail on tables because they misrecognize characters; that is not the case. Most failures are structural. When merged cells and multi-level headers are handled poorly, values end up under the wrong column. LightOnOCR-3's 4B model scored 83.9 on text, higher than dots.mocr's 82.5, but its table score was only 70.8—nearly 18 points lower. MinerU2.5-Pro scored 92.1 on tables, the strongest among open-source models, but its text score was only 77.4, a clear case of uneven performance.

Why not use olmOCR-bench? Datalab gave its reasons in the blog: in that benchmark, about 3–4% of the test ground-truth answers are incorrect or the rules are too subjective; the real ceiling is around 96–97%, and top models are already close to that ceiling. The remaining room for improvement comes from post-processing code written specifically for the benchmark. LightOn's table shows that Infinity-Parser2-Pro scored 85.9 raw and 87.6 with post-processing; LightOnOCR-3-4B scored 85.0 raw and 86.3 after normalization. Those post-processors are each about 700 lines of code. A one-point difference is larger than the gap between most models on the leaderboard.

Handwriting and non-English pages are the most obvious weaknesses of open-source models. chandra-ocr-2 scored 66.8 on handwriting, already the highest among open-source models; Gemini 3.8 Flash scored 79.4. Half of the nineteen models scored below 55 on handwriting.

The selection advice is actually quite clear: for the strongest overall performance, chandra-ocr-2, dots.mocr, MinerU2.5-Pro, and Infinity-Parser2-Pro are all within 3.1 points of one another. For tables, choose MinerU2.5-Pro and dots.mocr. For text and handwriting, chandra-ocr-2 leads. If you only have a CPU, use Indra OCR: 12.3M parameters, median page time of 357 ms on 4 cores.

In another reply, Vik said these scores largely match their "vibe test," and that dots.mocr and MinerU are indeed very strong. Someone asked whether Docling has been outclassed; he did not reply. Someone asked how OXR performs; he said he had not tried it yet and would take a look.

The credibility of a benchmark does not depend on who is first; it depends on whether the last-place result can be reproduced. Datalab open-sourced all adapters, documented each model's configuration in the blog, and put the dataset on HuggingFace. You may disagree, but you can run it yourself.

References:

• Niels Rogge tweet: https://x.com/NielsRogge/status/2114632691048214765
• Vik Paruchuri tweet: https://x.com/VikParuchuri/status/2114054449043751293
• Datalab blog: https://www.datalab.to/blog/open-ocr-models-omniparsebench
• GitHub repository: https://github.com/datalab-to/omni_parse_bench
• HuggingFace dataset: https://huggingface.co/datasets/datalab-to/omni_parse_bench