AI Reviewers May Reject 96% of Human Papers—The Problem Is Granularity

AI scientist systems have an often-overlooked critical flaw: when judging whether a research proposal is novel, they score it as a whole. One study put a number on it: existing judges may kill 96% of ideas that humans later published.

Novelty judgments at the whole-proposal level are hard even for human expert reviewers. Most papers do not have every contribution being new; they add increments on top of prior work. Once a single overall score is assigned, incremental ideas are easily misjudged.

Miri Liu, Xiayu Zhao, Encheng Xie, and ChengXiang Zhai at the University of Illinois Urbana-Champaign wrote a paper, “Do AI Scientists Know What's New?”, designing a framework to analyze novelty and using four datasets to construct proposal and review disputes to test the framework's effectiveness. Their proposed tool is called CLAIMNOV.

The idea is to change the granularity: break proposals down to the claim level, generate a claim profile that indicates which contribution is covered by which paper and to what extent, and then have the judge weigh this profile. This way, novelty checks can preserve incremental ideas built on prior work and directly tell the generator which claim to revise. It can even be used as a training reward for AI scientist agents.

On the data, ScholarEval's false-positive rate was 11.8%, while CLAIMNOV reduced it to 1.5%. AI Scientist v1's false-positive rate was 52%. CLAIMNOV retained only the core claims among 366 source names and did not keep NA (not extracted).

The poster Jinyan Su mentions seeing more and more research on “taste.” Judging novelty is inherently closer to a matter of taste than to plagiarism detection. Plagiarism detection can tell you whether two sentences look alike; taste has to tell you which sentence is worth keeping.