The leaderboard isn't out yet, but the rankings are already written

On X, Han Fang mentioned that he and @karthikabinav discussed a question that has long bothered them: how many model evaluations are just trying to match the vibe on Twitter?

Here, benchmark can be simply understood as an evaluation set or leaderboard used to compare model capabilities.

That phrasing is unflattering, but it is very concrete. Before an evaluation even appears, people already have a ranking in their heads. That ranking mostly comes from Twitter consensus. So when the results look wrong, the first reaction is to doubt the evaluation; the consensus itself is rarely examined.

Then comes filtering. Benchmarks that fit the narrative are trusted, retweeted, and written into discussions. Benchmarks that challenge the narrative quickly disappear. Many people see them and still don't want to pass them on.

The bigger problem is that a lot of so-called consensus comes from people who have only seriously used a handful of models. Their experience is limited, but their judgments are already fixed. When a new benchmark appears and happens to validate that fixed set of judgments, the loop closes. Then influential people stir it up a bit, and the echo chamber gets louder.

How much does this distort model evaluation? It is hard to quantify. But saying the impact is small is hard to believe.

For developers, looking at evaluation results requires one more layer of scrutiny: what is the task distribution, how is the sample quality, does the evaluation method have biases, and who is spreading the results? For ordinary users, leaderboards can be useful, but don't treat them as the final answer. Model reputation on social platforms is sometimes just a projection of usage habits and in-group preferences.

Evaluations are certainly useful. The trouble is that they are often used to rubber-stamp existing consensus.

Original tweet: Han Fang