Wink Pings

Compute Has Shifted, but Science Hasn't Caught Up: Six Open Problems in Post-Training

At frontier labs, post-training/RL now accounts for 55% of compute, while pre-training is down to just 7%. In a 51-page presentation at Cornell, Nathan Lambert laid out six open problems in post-training that no one has solved yet.

Nathan Lambert gave a talk at Cornell last week. It was a 51-page slide deck titled Pushing Frontiers in Open Language Model Post-Training. The core judgment: the center of gravity in compute has shifted from pre-training to post-training, but the scientific foundations of this field are far from keeping up.

First, look at how recipes evolved. In 2022, InstructGPT was a three-part stack: SFT → reward model → PPO. In 2024, Llama 3 and Tülu 3 added RLVR, and the recipe began to mature. In 2025, DeepSeek R1 made large-scale RL a central component: pure RL first elicits reasoning behavior, then SFT and RL are mixed. In 2026, MiMo-V2-Flash introduced MOPD: train multiple domain experts, then distill back into a single final model. Recipes changed more in the past year than in the previous three years combined. Roughly every two years a new wave of core techniques appears: RLHF and DPO in 2022–2023, RLVR and tool use in 2024–2025, and on-policy distillation in 2026–2027. No one yet knows what the next wave will be.

DeepSeek R1多阶段流程

MOPD:多教师on-policy蒸馏

The compute-shift numbers are even more direct. SemiAnalysis data shows that in Anthropic and OpenAI's compute allocation, pre-training fell from 67% in Q1 2024 to about 7% in Q4 2026, while post-training and RL rose from 5% to 55%. At the level of individual models, DeepSeek R1's RL compute was about 5% of pre-training, Beam exceeded 1.7×, and Inkling 2 Small was about 4×. Post-training is no longer a supporting actor.

Anthropic和OpenAI的算力分配

One direct consequence is that base model releases are becoming rarer. Among 12 open-weight labs, most have shifted over the past 12 months from releasing full base models to releasing only post-trained weights. A new concept follows: post-post-training—taking an existing reasoning model and adapting it to a domain directly, rather than starting from a base model. Harvey used Kimi K3 base for legal-domain post-training, letting law firms own their own models. Mercor and SkyRL trained Qwen3.5-397B with RL into a knowledge-work agent, and open-sourced both the scripts and the weights. This is turning into a business.

开源实验室的base model发布矩阵

Lambert listed six open problems he considers most important, ordered from low to high compute requirements.

1. Harness specialization and design

How to design environments and context management so that models work efficiently on specific tasks. This design space has received little serious exploration. His CLM (Context Language Models) is one direction: the model directly edits its own context rather than going through external harness commands. Someone in the comments mentioned PhantomEnvironments, which trains search agents in fictional worlds generated purely from rules; when transferred to real multi-hop retrieval benchmarks, it performs even better than training on real data. Work like this belongs in this quadrant.

2. Character training

How to train model character and values, and how this affects safety, trust, and diffusion. In industry, OpenAI is already writing a new version of the Model Spec for agents, while the open-source literature has not kept up. Open Character Training, which Lambert was involved in, used Constitutional AI to train the character of Llama 3.1 8B; models with different characters answer the same question in visibly different styles, but quantitative evaluation is extremely difficult. A commenter added that they would like to see more research on why models produce annoying AI-flavored language.

3. Open-source RL environments

Post-training research lacks the kind of public data resources that existed in the pre-training era. Good data projects often share one feature: the primary author read every single data point. The TMax project generated terminal-bench data through nine-dimensional task sampling, turning Qwen models from 2B to 27B into SOTA terminal agents; the complexity of its data pipeline far exceeds typical synthetic-data workflows.

4. Reasoning efficiency in RL

How to control token consumption and efficiency in reasoning chains. Common RL practice produces clear gains on easy problems, but the hardest problems barely move. The NGU (Never Give Up) idea is: if a batch of completions is all wrong, keep sampling. On Manufactoria, pass@1 on the hard test rose from 19% to 88%. Fairness of experiments is extremely hard to guarantee, but the direction is clear.

NGU vs GRPO

5. Scaling laws for RL

Pre-training has ladder methods for predicting the performance of large-scale runs; RL has no equivalent. ScaleRL uses a sigmoid to fit the relationship between reward and log compute; it is a start, but still far from a usable methodology.

6. Multi-agent scaling laws

Multi-agent systems deserve serious study: which tasks actually benefit from parallel speedup? How should model diversity be used? What needs to change during training for sub-agents? OpenAI's GPT-5.6 and Anthropic's Claude Opus 5.5 both show that more agents can reach higher scores faster, but this is mostly empirical.

The slides end by discussing the impact of RSI (recursive self-improvement) on the research ecosystem. Anthropic's R&D automation index shows that the share of R&D tasks led by Claude rose from less than 1% in February to 26% in August. But Lambert's view is: LLMs accelerate output, not understanding. Lambert says he writes less code, but his busyness as a scientist has not changed. Young researchers doing inefficient work by hand is often the only way to build understanding.

A historical footnote: the DPO paper was released in May 2023, while Zephyr β and Tülu 2 did not appear until October and November. Everyone was trying the method, but no one shipped it first. Chris Manning later thanked Lambert for saving DPO with those two models. Work that applies an already-known approach would now be much faster. But work involving organizational complexity, such as Tülu 3, Olmo 2, and Olmo 3, has not changed pace.

Lambert treats the six open problems as the research direction for Trillium Labs; the slides themselves are a statement of goals. Compute has shifted; science has not caught up. This is probably the battlefield most worth watching in the next few years. The full slides are at natolambert.com, and the video will be re-recorded soon.

发布时间: 2026-11-22 18:40