Wink Pings

OpenAI Discloses Pretraining Data: Does One Number Decide AI Capability?

OpenAI Chief Scientist Jakub Pachocki has unusually made public internal pretraining perplexity data and proposed using it as the basis for a shared safety ceiling. From 0.99 to 0.76, each number corresponds to a leap in AI capability.

OpenAI Chief Scientist Jakub Pachocki published a long essay titled "Perplexity and Intelligence." He did two things: he made public pretraining perplexity data that OpenAI has tracked internally for years, and he proposed that the entire industry establish shared safety standards based on this metric.

Pachocki's core argument is that modern AI capability essentially comes from a simpler process: training a compressor (equivalently, a predictor) to predict the next token on ultra-large-scale data. Various prompts, fine-tuning, and reinforcement learning merely 'invoke' the intelligence already present in this compressor.

Perplexity measures how accurate that prediction is. The lower the number, the more accurately the model predicts the data. OpenAI tracks log-perplexity, also called cross-entropy loss. Starting with GPT-3.5, they have recorded this number on the same internal codebase evaluation set.

These numbers correspond to capability transition points:

  • GPT-3.5, about 0.99: fluent conversation, the starting point of ChatGPT.
  • GPT-4, about 0.91: able to effectively use chain-of-thought reasoning, giving rise to modern reasoning models.
  • Below 0.8: transformative capabilities begin to emerge in cybersecurity and mathematics.
  • About 0.76: begins solving long-standing open problems in mathematics.

image

Pachocki describes a scenario: after the model was upgraded to 0.76, it solved, one after another in a Slack thread, multiple problems that he and other mathematicians had long pondered. "It felt surreal," he said.

He speculates that the field of deep learning research could also see a similar avalanche-like advance in 2027. Research on pretraining algorithms may experience a sudden burst, following a different path from gradual optimization. He specifically notes that AI has already made meaningful contributions to its own design, but he believes progress in alignment and monitoring may not be fast enough to support a full acceleration of pretraining.

This is also why he proposes "common limits." Specifically: set a ceiling based on metrics such as perplexity, with developers committing not to cross that limit until they reach a specific milestone in understanding AI behavior (for example, developing a successor to CoT monitors).

Disclosing internal data comes at a cost: competitors will use it to see where OpenAI stands. But Pachocki believes the urgency of establishing shared safety standards outweighs this concern. He hopes that before the first deployment of frontier models, developers can jointly establish standards for evaluation specifications and red-team safety cases.

This proposal is not without holes. Pachocki himself lists four challenges: different labs use different datasets, making cross-lab comparison difficult; perplexity does not directly correspond to any specific task capability; progress outside pretraining, such as chain-of-thought, cannot be reflected in this metric; and no one knows the lower bound of the inherent entropy of data. The closer one gets to that lower bound, the larger the intelligence leap corresponding to the same decrease in the number.

These limitations make the proposal look less like a perfect solution and more like an actionable starting point. It moves the safety discussion from 'feel' to 'measurement.' First there is a dashboard that the whole industry can see together; then we can talk about when to hit the brakes.

The question now is: will other labs follow suit and disclose their data? And if the 2027 prediction comes true, will this dashboard still have time to sound the alarm?

发布时间: 2026-11-23 03:46