Stanford compresses building an LLM from scratch into 2 hours, but the workload is 10x that of a regular course

Stanford has released a 2-hour video course with a straightforward title: Building an LLM from Scratch. It covers tokenization, how a model decodes your prompt, the training pipeline, and building the architecture from scratch. The video is online; in the comments, someone immediately bookmarked it, someone said the decoding section is worth watching first, and another person said that after watching the tokenization part, knowing how byte-pair encoding works under the hood changed their understanding of the context window.

But there is an easily overlooked fact here. The 2-hour video is not the full course; it is a condensed version of CS336. CS336 stands for Language Modeling from Scratch and is taught at Stanford by Percy Liang and Tatsunori Hashimoto; the course website puts the lecture notes, recordings, and assignments for every class online. The video is just the entry point—the real work comes afterward.

First, look at the course requirements. Students need to be proficient in Python, have experience with deep learning and systems optimization, understand calculus, linear algebra, and probability/statistics, plus basic machine learning. This is a 5-unit course, and the official wording is 'very implementation-heavy,' meaning the amount of code is enormous. The course page states clearly: students receive very little scaffolding, and the code they have to write is at least an order of magnitude greater than in other AI courses. In other words, the 2-hour video teaches you to understand the big picture, while the assignments teach you to write it yourself.

Here is a breakdown of the five assignments:

Assignment 1, foundations. Implement a tokenizer, the Transformer architecture, and an optimizer, then train a minimal language model. In the comments, someone joked that after watching the 2-hour course, their tokenizer still looked like a confused raccoon. This is not just a joke: Assignment 1 is about getting that raccoon to learn to read.

Assignment 2, systems. Use profiling tools to measure model and layer performance, implement FlashAttention2 yourself with Triton, and then rewrite the training code into a memory-efficient, distributable version.

Assignment 3, scaling. Understand the role of each Transformer component, call the training API to fit scaling laws, and predict how the model will perform as scale increases.

Assignment 4, data. Turn raw Common Crawl dumps into usable pretraining data, perform filtering and deduplication, and examine how these operations affect model performance.

Assignment 5, alignment and reasoning RL. Do supervised fine-tuning, use reinforcement learning to train the model to solve math problems, and optional parts also include safety alignment methods such as DPO.

Self-learners following along cannot avoid GPU bills. The course page offers several options, with public pricing for a single B200: Modal at $6.25 per hour, with $30 in free credit per month and billing by actual compute usage; Lambda at $6.69 per hour; RunPod at $4.99 per hour; Nebius at $5.50 per hour, or $3.05 per hour for spot instances; Together at $7.49 per hour, with a minimum of 8 GPUs. The official recommendation is to first debug correctness on CPU, then run training and benchmarks on GPU.

Image: Stanford NLP Group and CRFM logos provided on the course page

The course specifically addresses its stance on AI tools in the honor code: you may use ChatGPT to ask low-level programming questions or high-level concepts about language models, but you may not use it directly to solve problems. They also strongly recommend turning off autocomplete tools such as Cursor Tab and GitHub Copilot, for a very practical reason: AI autocomplete makes it harder to deeply understand the material.

Back to that 2-hour video. It is good for building intuition and understanding which steps an LLM goes through from text to output. One commenter said the 2 hours pry open the black box, and that's true. But after the black box is opened, you have to build one yourself. Stanford made the entry point low enough; the barrier is left in the assignments. After watching the video, if you want to keep going, start by writing the tokenizer.

Course website: https://cs336.stanford.edu
YouTube playlist: https://www.youtube.com/watch?v=JuoVZkPBiKk&list=PLoROMvodv4rMqXOcazWaTUHhq-yembLCV