Karpathy's Free AI Course: Don't Start with nanoGPT

Karpathy has a set of AI courses on GitHub ordered by dependencies, 7 repositories, and a screenshot of a tweet shows a total of 306,741 stars. The tweet reposted by Machine Learning FLX calls it a free AI degree. Many people bookmark it, few run it in order. The difference isn't that it's free; it's the order.

The path in the tweet starts with backpropagation and ends with an agent that can run experiments overnight. Each repository is a prerequisite layer for the next. Below I break it down in the original order, keeping the links and data.

1. micrograd: how neural networks learn
   Implement backpropagation in about 100 lines of Python, then stack on an approximately 50-line neural network library with a PyTorch-like API, but operating only on scalars. It breaks each neuron's additions and multiplications into a computation graph. The repo demo uses a 2-layer MLP for binary classification on the moon dataset.
   
   
   Link: https://github.com/karpathy/micrograd

2. nn-zero-to-hero: video course and notebooks
   Karpathy's YouTube course, from a single neuron to GPT. The Jupyter notebooks written in the videos are in the lectures directory. The course covers makemore, MLP, BatchNorm, manual backpropagation, WaveNet-style structures, GPT, and tokenizer. It is well suited for extending the scalar backpropagation from repo 1 to tensors.
   Link: https://github.com/karpathy/nn-zero-to-hero

3. minbpe: how text becomes tokens
   A minimal implementation of byte-level BPE. Models such as GPT, Llama, and Mistral all use BPE to train tokenizers. The repo includes BasicTokenizer, RegexTokenizer, and GPT4Tokenizer, the last aligned with OpenAI tiktoken's output. You can also train a tokenizer on your own text. Karpathy mentioned in the course that some odd LLM behaviors can be traced back to the tokenization layer.
   Link: https://github.com/karpathy/minbpe

4. nanoGPT: train your own GPT
   The tweet calls it the simplest GPT training and fine-tuning repo. By this point, tokenizer, backpropagation, and Transformer architecture have all appeared in the first three layers, and nanoGPT assembles them into a trainable model.
   Link: https://github.com/karpathy/nanoGPT

5. llm.c: laying the training process bare
   The same training process, written in raw C and CUDA, with no PyTorch underneath. It is good for seeing what the framework hides for you. The tweet places it after nanoGPT, with a clear intent: first know how to use it, then look at the low level.
   Link: https://github.com/karpathy/llm.c

6. nanochat: an end-to-end chat model
   A complete training pipeline for a chat model. The tweet gives a cost comparison: GPT-2-level training cost about $43,000 in 2019, versus about $48 here. This figure is according to the tweet and is not a general quote, but it shows how much training costs have dropped.
   Link: https://github.com/karpathy/nanochat

7. autoresearch: let an agent do research
   The agent modifies the training code itself, trains for 5 minutes, keeps improved versions, and then repeats, running overnight. The tweet places this repo last because the previous six layers provide the code it needs to modify, the model it needs to evaluate, and the logs it needs to read.
   Link: https://github.com/karpathy/autoresearch

The tweet ends bluntly: train a GPT from scratch, and you'll become the person AI projects get handed to, along with a raise. That line has a recruiting-ad feel. The repos themselves are another matter: they hard-code the dependency relationships of a learning path. You can skip around and get demos running, but when you hit an error, you still have to go back to the previous layer. Free AI courses are not scarce; what's scarce is running one loop through in order.