OpenAI Wants to Use Perplexity to Set a Ceiling on AI Development, but Is This Ruler Accurate Enough?
OpenAI chief scientist Jakub Pachocki has publicly disclosed internal perplexity data and proposed using it as the basis for setting a safety ceiling for the entire AI industry. Can a single metric bear that weight? Will other labs follow suit?
OpenAI chief scientist Jakub Pachocki published a long article on the official website titled "Perplexity and Intelligence". In it, he publicly disclosed the metric OpenAI has long used internally to measure pretraining progress: perplexity.
Perplexity measures a model's ability to predict the next token; the lower, the better. Pachocki's observation is that AI capability comes from a simpler, less interpretable process: training a compressor on an enormous corpus. As perplexity decreases, capabilities emerge. He has previously called this kind of capability an "alien mind."
On the dataset OpenAI has tracked internally for a long time, GPT-3.5's log-perplexity (cross-entropy loss) is about 0.99, which is the foundation of ChatGPT. GPT-4 is 0.91 and begins to use chain-of-thought effectively. When loss drops below 0.8, models show leap-like capability jumps in cybersecurity and mathematics. At around 0.76, models begin to solve long-standing open mathematical problems one after another.
Pachocki argues that there is a relationship between capability emergence and perplexity, but it is impossible to predict in advance at which threshold a specific capability will appear. In other words, this metric can provide early warning, but it does not tell you what will be unleashed.
He proposes that the entire industry should establish a common safety floor based on metrics such as perplexity. Developers would commit not to cross this line until they reach a specific milestone in understanding and monitoring AI behavior. For example, building the next generation of CoT (chain-of-thought) monitors. He also predicts that AI's acceleration of its own development will soon spill over into pretraining algorithm research, and that an "avalanche-like" breakthrough similar to the one in mathematics may be seen in 2027. If pretraining progress accelerates, perplexity will fall faster, and the significance of the safety ceiling will be greater.
The proposal sounds reasonable but is hard to implement. Different labs have different datasets and data mixtures, so the numbers cannot be compared directly. Pachocki himself acknowledges this. In addition, perplexity does not directly correspond to any specific task; chain-of-thought reasoning significantly improves capabilities without lowering perplexity. It alone is not enough.
More noteworthy is that OpenAI chose to make this internal metric public. This is tantamount to telling competitors how far its pretraining progress has gone. Pachocki says that the urgency of establishing shared safety standards matters more than competition. Whether this rationale holds up remains to be seen.
Perplexity is not a universal ruler, but it provides a rare dimension for horizontal comparison. As AI capabilities become increasingly difficult to measure with benchmarks, a simple number may help us see progress more clearly. OpenAI has taken the first step; next, it remains to be seen whether other labs will follow.
发布时间: 2026-11-23 04:47