11 Hours to Take Apart a Reasoning Model, 90 Files to Take Apart LLM Architectures

Sebastian Raschka has released his 'Building a Reasoning Model from Scratch' course: 8 videos, 11 hours. Around the same time, someone named Lavish Gupta open-sourced a repository containing PyTorch implementations of 90 modern LLM architectures, with only one model.py file per model.

The two efforts are not directly collaborative, but they point to the same approach: break models down into code that can be read and run.

Starting from a pretrained Qwen3 0.6B

Raschka's course does not start by training an LLM from random weights. He directly loads the pretrained weights of Qwen3 0.6B, then implements evaluation, inference-time scaling, reinforcement learning, and distillation on top of that. The reasoning is straightforward: training a base model is the subject of another book; here the focus is on reasoning methods.

The course prerequisites are Python and PyTorch. The early text generation and evaluation examples can run on a CPU; the later training experiments are best with a GPU. The repository provides hardware and environment recommendations.

He drew an engine analogy diagram that places a regular LLM, a reasoning model, and an agent harness in relation to one another. The model is the engine, the agent harness is the mechanical structure around it, responsible for context management and tool calls. A reasoning model is an upgraded engine, with reinforcement learning using verifiable rewards added, and optionally inference-time scaling on top. Most new models now come with a reasoning mode built in; this diagram explains where each one sits.

The eight video titles, in order, are:

1. Motivation and code environment setup
2. Loading the base model, text generation, KV cache
3. Building a verifier for evaluation and reinforcement learning
4. Inference-time scaling: sampling and self-consistency
5. Inference-time scaling: scoring and self-correction
6. Reinforcement learning: implementing GRPO
7. Reinforcement learning: training tracking, clipping the policy ratio, KL term
8. Distilling a reasoning model

The verifier built in episode 3 is not only used for running MATH-500 evaluation; in episode 6 it also becomes the reward signal for RLVR. This design makes the series not eight isolated demos but a single line: first there is a verifier that can score, then it is used as a reward to train the model.

The companion book is 'Build a Reasoning Model (From Scratch)', available from Manning and Amazon, with a color print edition. Raschka's suggested way to learn: read a chapter, follow the corresponding video to write code, then do the exercises.

90 Architectures, One File Each

Gupta's repository works at a different layer. He wrote 90 LLM architectures as standalone PyTorch files, not relying on transformers model code. Each folder contains model.py, the official config.json, tokenizer.json, and in some cases an architecture diagram.

The covered families include LLaMA 1 through 4 Maverick, the dense and MoE versions of Qwen1.5/2.5/3, DeepSeek V2/V3, GLM-4.5 through 5.2, Gemma 1 through 4, GPT-OSS 20B/120B, Mistral Small and Large, MiniMax M2 through M3, and several Cohere models. In addition, 20 models have only a config and tokenizer, with implementations not yet written. There are also 9 standalone attention mechanism files, from Bahdanau additive attention to sliding-window attention.

The goal is not benchmarking; it is to make the differences between architectures visible. The same building blocks—RoPE, RMSNorm, SwiGLU, GQA—are recombined in different models, with newer additions such as sliding-window layer patterns, logit soft-capping, QK-norm, YaRN, and sparse MoE routing. You can read one file from start to finish without jumping through inheritance chains.

One thing about this repository deserves separate mention: it clearly documents what it has not finished. The README explicitly states that all demos use randomly initialized weights, the generated text is meaningless, and they only verify that the architecture can be built and the forward pass can run. There is no code for loading real pretrained weights. A logit-level comparison script was run but not committed to the repository, and there is no CI.

A list of known limitations is given, including: Gemma 3 1B's forward pass errors out directly because self.emb_scale is undefined; Qwen3 4B/8B/32B is missing QK-norm; the Q/K/V bias configuration for Qwen1.5 and Qwen2.5 is wrong; and the multi-token prediction module and vision and audio encoders are not implemented.

Writing clearly about what is unfinished is more useful than pretending everything runs. Contributors know where to start, and users know not to treat it as production code.

There is also a small closed loop: the architecture diagrams in the repository come from Raschka's Architecture Gallery, as noted in the acknowledgments. Raschka's book and course explain the principles, while Gupta's repository lays out the architectures; the readers on both sides are the same group of people.

After Taking It Apart

In recent years, reasoning models have been described in almost mystical terms. Taken apart, they are just a layer of post-training added to a regular LLM, using reinforcement learning with verifiable rewards, and optionally inference-time scaling on top. Gupta's 90 architectures, for all their variation, are still arrangements and recombinations of those building blocks.

But 'taking it apart' is not the same as 'reproducing it.' Gupta's repository itself says that running with random weights does not mean the architecture is correct, and the logit comparisons have not yet been added to automated tests. Raschka's course also starts from a pretrained model; it does not teach you to train a usable model from scratch. Both are at the level of understanding, not leaderboard chasing.

Questions in the comments focus on where to start and whether the hardware is sufficient. The course design gives part of the answer: for a 0.6B model, early generation and evaluation can run on CPU, while training needs a GPU. Doing RLVR experiments on consumer-grade hardware is not impossible, but do not expect to reproduce the curves from the paper.

Raschka says he will take a break for a while and come back in November with LLM model and research updates. The course, book, and code remain available. Gupta's repository is still accepting PRs, and what it welcomes most are verification reports and implementations of the models marked with 🚧.

If you want to get hands-on, starting from Raschka's second episode (loading Qwen3 0.6B and adding KV cache) or from the smallest Qwen3 0.6B folder in Gupta's repository is more useful than reading another 'reasoning model survey'.