OpenAI publicly discloses pretraining perplexity for the first time, aiming to install a speed limiter on the AI race
OpenAI Chief Scientist Jakub Pachocki makes a rare disclosure of internal pretraining data and proposes using perplexity as an industry safety baseline. This obscure metric could become a new checkpoint in the frontier model competition.
On November 22, OpenAI Chief Scientist Jakub Pachocki posted a long article, unusually disclosing internal pretraining progress. The article is titled Perplexity and Intelligence. There are two core moves: publishing the perplexity data OpenAI has tracked for years, and proposing that the entire industry use perplexity as a common safety-limiting metric.
First, let's clarify the concept. Perplexity measures a model's ability to predict the next token. The lower it is, the stronger the model's ability to compress data. Pachocki argues that modern AI is essentially training a compressor to make predictions over massive corpora, and various capabilities later emerge from this predictive ability.
Since the GPT-3.5 era, OpenAI has been measuring log-perplexity—that is, cross-entropy loss—on a fixed internal codebase dataset. Several key milestones:
- GPT-3.5, with log-perplexity around 0.99, could converse fluently and gave rise to the first generation of ChatGPT.
- GPT-4, around 0.91, could effectively use chain-of-thought for reasoning.
- Now, after loss dropped below 0.8, models began to show transformative capabilities in cybersecurity and mathematics.
- When log-perplexity reached around 0.76, models began to solve long-standing unsolved math problems in succession.
The most counterintuitive part of this observation is that capability emergence corresponds to specific perplexity thresholds, but no one could predict it in advance. Pachocki himself says these capabilities are essentially latent abilities that emerge alongside improvements in compression rate.
In another post, someone did the math based on this number. OpenAI's internal model has an average predicted probability of about 48% for the next token in its own codebase; at 4 bytes per token, that amounts to a 30:1 compression ratio. That number is astonishing enough.
Pachocki's disclosure this time is not simply a show of results. He explicitly says that publishing internal perplexity will let competitors see exactly where OpenAI stands, but the urgency of establishing industry-shared safety standards outweighs that concern.
His proposal is: frontier model developers jointly set a perplexity ceiling and do not cross that line until the ability to monitor and understand AI behavior reaches a specific milestone. It can be understood as installing a speed limiter on the pretraining race.
He also offered a more urgent timeline. AI's participation in its own algorithm research could bring about sudden shifts; similar turning points have already appeared in mathematics. He believes that as early as 2027, pretraining research could see avalanche-like progress.
Of course, perplexity as a public metric has many problems: different labs use different datasets, making direct comparison difficult; perplexity cannot be mapped to a specific task; reasoning techniques such as chain-of-thought do not lower perplexity but can greatly improve capabilities; and no one knows the entropy floor of the data itself—when approaching that floor, small improvements may correspond to large leaps.
Even so, Pachocki believes perplexity is one of the few metrics that has stood the test of time and can be discussed across labs. OpenAI has taken the first step; next, it remains to be seen whether other labs follow suit.
In the comments, someone joked that they finally understand why Codex loves using the word "gated" so much. Others are waiting to see whether other labs will follow. This question is indeed more worth watching than the models themselves.
发布时间: 2026-11-23 02:39