AI Proposed 135 Research Questions; Experts Recognized Only One Third

Fan Nie and team did a project called After42, letting AI propose research questions. The name is borrowed from The Hitchhiker's Guide to the Galaxy: a supercomputer calculates the ultimate answer as 42, but no one knows what the question is. After42 wants to reverse this and have AI find questions worth asking.

29 experts participated in the evaluation, covering mathematics, statistics, theoretical computer science, and physics. Of the 135 final questions, 45 were judged genuinely interesting and worth doing. About one third.

But this was the result after iteration.

In early experiments of the same pipeline in the phase transitions subfield, experts judged only 3% of initial proposals as 'interesting.' After filtering and multiple rounds of solving-and-refining, the final questions rose to 33%.

Solve first, then ask

The core idea is not complicated: good questions often become clear only in the process of trying to solve them.

Researchers give a broad field goal, such as 'understand how large numbers of local random rules produce macroscopic phase changes.' The system first generates a large batch of candidate questions from the literature, filters out weak ones, and sends the rest into multiple rounds. In each round, a solver agent tries to solve the question, a developer agent revises the question based on findings, and model judges compare old and new versions. A revision passes only if all judges approve.

An example illustrates this process. For the sparse geometry question, the initial question was: can geometric information be recovered from queue-length records of breadth-first search? Attempts to solve it found that the number of triangles was key, and most of these triangles fell outside large components. The question was finally rewritten as: for random geometric graphs on high-dimensional spheres, at a specific dimension scale, is it equivalent to an Erdős–Rényi graph plus a Poisson number of random triangles? Experts commented that this rewrite 'corrected the initial intuition about where geometric information hides.'

The curve from 3% to 33% looks like this:

97% of questions are just permutations and combinations

The paper uses the word 'combinatorial' to mean questions that build directly on known work, reorganizing, modifying, or taking a next step along existing objects. In the mathematics sample of 100 initial proposals, 97 belonged to this category. Of the 70 final questions, 61 still did.

Some commenters noted that the 97% figure sounds like a model limitation, but looking back at human research, most questions are also combinatorial. Daniel Lenton said his own PhD work was 97% combinatorial too. That makes sense. However, the paper mentions a distinction: Broadbent and Hammersley proposed the percolation model in 1957, and Erdős and Rényi studied the evolution of random graphs in 1960. Questions of this kind introduce new objects and open new directions, and they are almost absent from current AI proposals.

Questions experts find interesting can also be combinatorial. What is truly scarce is the part that breaks out of combinatorial thinking.

Models can capture consensus, but not individual taste

The second question is whether AI can judge which question is interesting. Four frontier models and nine experts ranked the same batch of 70 questions.

The top 20 of the model ensemble covered 80% of the experts' consensus top 20; this number is similar to the alignment between individual experts. But for an individual expert's top 20, the model ensemble covered only 35% to 85%. Questions ranked highly by only one expert were almost never selected by the models. The paper notes that such questions sometimes precisely open new directions.

Research taste is dispersed, and experts disagree on many questions. Models can capture consensus, but outside consensus they have no ability to judge.

Great variation across fields

In the phase transitions subfield, 47% of final questions were considered interesting; black hole ringdown, 38%; online algorithms, 33%; prediction-powered inference, 14%; plasma physics, 0%. The closer a subfield is to mathematics, the higher the proportion.

The mathematics round ran with GPT 6 Astra for a day, producing 70 final questions and consuming about 260 agent-hours and 2.6 billion tokens.

Questions are first-class citizens

When Han Xiao retweeted it, he said the smartest part of the design is that solving is not the goal: solved questions get filtered out as 'medium,' while unsolved questions survive. The question itself is a first-class citizen. This is consistent with the logic of curiosity-driven systems.

Someone asked whether he thought best practice was question generation or evaluation. He replied that verifiers are a separate topic; an efficient, safe, and hard-to-game sandbox verifier is important, but good software engineering can solve it. After42 focuses more on asking questions itself.

The paper also discusses the limitations of evaluation: there is no reliable automatic evaluator, final judgment depends on experts, and expert resources are limited. The number of questions, subfields, and experts covered by the whole evaluation is not large.

A few observations

What is noteworthy about this work is that it does not hard-code 'interesting' into an optimizable proxy metric. Metrics such as citation counts and proof length are easy to optimize, and also easy to drift away from what researchers truly care about. After42 leaves final judgment to experts, and models are only responsible for narrowing the candidate range.

The cost is high. On average, experts spend 35 minutes reading and writing comments for each question. This process cannot easily be scaled.

The bottleneck for AI question-asking is not generation but judgment. The approach After42 offers is to let researchers retain the final choice, with models helping filter at the consensus level. This may be the most honest approach currently available.