Wink Pings

Build GPT by Hand from Scratch: This Open-Source Repository Has Paved the Way

The official code repository for Build a Large Language Model (From Scratch). Implement GPT from scratch with PyTorch, covering pretraining and fine-tuning, and it can run on an ordinary laptop. It includes a video course and many advanced implementations; if you want to deeply understand LLM internals, this is one of the paths worth bookmarking.

There’s a GitHub repository worth bookmarking: the official code for Sebastian Raschka’s Build a Large Language Model (From Scratch). It contains all the code for the entire book, from data preprocessing to pretraining and fine-tuning, implementing a GPT step by step with PyTorch.

Build a Large Language Model (From Scratch) cover

Many people hit a wall after working with LLMs for a while: they can use APIs but don’t know what happens inside the model. This repository offers a path that sounds silly but is actually the most direct: write it yourself. The author lowers the barrier: the code does not depend on external LLM libraries, and the main chapters can run on an ordinary laptop. If you are new to PyTorch, Appendix A provides a concise primer.

The repository’s chapter arrangement is clear:

  • Chapter 2: Text data processing, including implementation of a BPE tokenizer
  • Chapter 3: Attention mechanisms
  • Chapter 4: Implementing a GPT model from scratch
  • Chapter 5: Pretraining on unlabeled data
  • Chapter 6: Fine-tuning for text classification
  • Chapter 7: Fine-tuning to follow instructions

Full-book knowledge framework diagram

Each chapter has complete code and exercises, with answers in the appendix. If you don’t want to read the book, there is also a 17-hour 15-minute video course in which the author types out the code chapter by chapter. In addition, each chapter comes with a PDF of about 30 quiz questions, free to download and suitable for self-testing.

Video course screenshot

The bonus section is even more worth seeing. Besides the standard GPT, the repository also includes from-scratch implementations of architectures such as Llama 3.2, Qwen3, and Gemma 3, as well as advanced topics such as LoRA fine-tuning, DeepSeek sparse attention, and Mixture-of-Experts. After finishing the basic chapters, these materials can broaden your perspective.

This repository can also serve as a reference manual. When you encounter a concept you don’t understand while coding, go to the notebook for the corresponding chapter and look it up; it’s faster than digging through docs or a search engine.

The author also wrote a sequel, Build A Reasoning Model (From Scratch), which starts from a pretrained model and implements methods such as inference-time scaling, reinforcement learning, and distillation. If you’re interested in the reasoning direction, you can continue with that.

Repository address: https://github.com/rasbt/LLMs-from-scratch
If you plan to get hands-on, clone it directly:
git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.git

发布时间: 2026-11-21 14:36