109M Parameters Beats 72B: LeCun's Paper Is Pulling the Rug Out From Under Large Models

Yann LeCun and Randall Balestriero recently published a paper. The tweet sums it up bluntly: 109M parameters beats 72B. LeCun has argued for years that scale is not the key to making models understand the world, while Meta bet on the other side. The tweet says he left Meta in November, opened his own lab in Paris, and last Thursday published this proof.

Since 2024, a popular hypothesis has circulated in the industry, later often called the Platonic Representation Hypothesis. As long as a model is trained at sufficient scale on text, images, and audio, its internal representations will converge on the same representation of reality. Scale will solve everything. This hypothesis has always lacked a proof.

Balestriero and LeCun turned it into a theorem. The abstract states that JEPA can learn the same representation across any number of modalities, even when the positive pairs are merely different views of the same scene, with bounded mismatch. What really determines learning efficiency is whether the two things shown to the model are two views of the same thing. JEPA takes the joint-embedding prediction route, having the model learn to align two views in representation space rather than relying on word-by-word prediction.

A photo and its caption are two views of the same thing. Two random paragraphs from the same document are not. Next-token prediction falls into the latter category. Nearly all chatbots run primarily on this objective.

In tests, the first method reached equivalent understanding with 21.6M samples. Next-token prediction needed 900M. A 109M-parameter text model ended up closer to the image model's worldview than a 72B-parameter language model. 660 times smaller.

The abstract also mentions validation from 145k captions up to 940M web image-text pairs, yielding an alignment law that can predict experiments not yet run. Good views reach the same alignment with 42 times fewer samples. A 109M text JEPA aligns better with DINOv2 and CLIP image encoders than a 72B text encoder. On ImageNet-100 five-way zero-shot accuracy it hits 64 percent, versus a top of 56 percent for language models. In the figure, the blue line is good-view JEPA, the orange line is the next-token LLM objective, the red line is bad-view JEPA, the x-axis is the number of training samples, and the y-axis is alignment with DINOv2.

What stings most about these numbers is that over the past two years the industry has bet most of its chips on parameters and compute. Scale is easy to talk about, easy to raise money on, easy to stack GPUs for. Data pairing and objective functions are hard to explain and hard to turn into a launch event. If the paper holds up, large models will not immediately become obsolete. But budget spreadsheets will need recalculating: for the same alignment, a 42 times difference in sample efficiency and 660 times fewer parameters. Where should the money and GPUs go?

Original tweet: https://x.com/ylecun/status/2116977874913648878