Can a small model beat GPT-4 with 10,000 samples? Stanford releases its entire self-improvement course

All 9 lectures of Stanford CS329A Self-Improving AI Agents are now on YouTube. The course homepage and playlist are public, and the course goal is very concrete: build an agent that can improve itself. The backbone is frontline experience plus close reading of papers; both instructors have worked on PaLM, Gemini, and Claude.

The course thread is not complicated. The 9 lectures revolve around one loop: sampling -> verification -> selection -> training -> a stronger model. When watching, use this line as an index, mapping each lecture to which step it corresponds to, so it is less easy to lose the thread. The course homepage is here, and all videos are in the YouTube playlist.

The question on the poster is straightforward: can a small model beat GPT-4? The label says LLAMA 3 8B, with 10,000 samples per question and a verifier. At the bottom there is also an English slogan: LARGE LANGUAGE MONKEYS · REPEATED SAMPLING. This image is more interesting than an abstract discussion: when sampling is piled up to 10,000 times, can the verifier actually filter out the answer?

Why have coding agents become widespread first, while writing still can't? The discussion keeps coming back to verification. One commenter pointed out the generation-verification gap: code matures first, and it has little to do with whether models prefer code. Tests, compilation, and execution can provide low-cost, repeatable feedback. For creative writing to replicate this flywheel, what is missing is a calibratable verifier; no amount of additional sampling can make up for it. Another person added: sampling and verification for code are simple because the compiler is the judge. For prose, the judge is another model, and that model itself is not very good at writing.

There is also a more practical warning. One commenter said the easiest place for self-improvement to fool itself is verification cost. The weaker the verifier, the faster the loop spins, and the more the model believes it has become stronger. First bring down the cost of verification, then talk about the 9-lecture loop; otherwise it is just spinning in place. Evaluation sets also have pitfalls. One commenter suggested setting aside a group of tasks that do not participate in selection and training, and testing on them at the end. Beyond higher scores, you also need to see whether the improvement holds up on problems that have not been repeatedly polished.

meng shao also made a demo video to quickly go through the course skeleton. He mentioned that he has recently been refining a set of demo video Skills, intended for courses, open-source projects, product launches, and article expression, and will release them once he is satisfied. For readers who do not write code, you can also first have AI explain the core of each lecture in plain language, then go back and watch the original videos; it will be less intimidating to work through.

The course is suitable for working through slowly on weekends. The original tweet is here.