Where do large-model training and inference hit bottlenecks first? This 230-minute course goes from VRAM to serving

Jia-Bin Huang has put a series on YouTube, 9 episodes, about 230 minutes, on the theme of LLM Training and Inference. The course is still WIP, and the author himself asked on X what else should be added.

Video playlist: https://www.youtube.com/playlist?list=PLXrQ2OzD3YKA

The order of this course is worth watching. It does not start from parameter scale; it starts from the model not fitting on one GPU. VRAM is insufficient first, and data parallelism then replicates optimizer states and gradients, so ZeRO and FSDP come in, and then it moves toward thousands and tens of thousands of GPUs. This order is closer to the engineering reality.

The 9 episodes roughly cover the following:

1. ZeRO & FSDP, about 23 minutes. How floating point and mixed precision consume VRAM; data parallelism replicates optimizer states and gradients. ZeRO-1/2 shards optimizer states and gradients, ZeRO-3 further shards weights; FSDP is the same sharding schedule implemented module by module in PyTorch.

2. Five-dimensional parallelism, about 40 minutes. When pure data parallelism is not enough, split along data, tensor, sequence/context, pipeline, and expert dimensions. Tensor parallelism's row/column splitting and all-reduce; context parallelism and ring attention; pipeline from GPipe, 1F1B to zero-bubble / DualPipe; MoE's all-to-all; and using Llama 3's 16384-GPU layout as an example, it gives a three-step cheat sheet for arranging parallelism.

3. LoRA and variants, about 21 minutes. Freeze W0 and learn low-rank ΔW=BA; initialization commonly uses random A and zero B. LoRA+ gives A/B different learning rates; QLoRA uses NF4 etc. to quantize the backbone while keeping adapters at high precision, and the subtitles say it can fine-tune 70B-scale models on a single 48GB GPU—this is the speaker's statement; VeRA shares a frozen random basis and only trains scaling; DoRA separates magnitude and direction.

4. Speculative decoding, about 25 minutes. A small draft model guesses ahead, and the large model verifies in parallel; rejection sampling guarantees the same distribution as the target model. Then it expands on draft tree, Medusa, MTP, the EAGLE series, and DFlash / DFlash 2.

5. On-policy distillation, about 31 minutes. From soft labels and dark knowledge to sequence-level distillation; exposure bias; on-policy RL rewards are sparse. The student uses its own trajectories but still gets dense guidance from the teacher; emphasizes the KL direction; applications mentioned include reasoning, personalization, data reuse, and self-distillation.

6. GRPO, about 23 minutes. From REINFORCE, baseline and value functions, GAE, Actor-Critic, TRPO/PPO—especially clip—to GRPO: sample a group of answers for the same question, normalize rewards within the group as advantages, share the advantage across the whole segment, and optionally avoid training a separate value model; add KL relative to the reference policy. Dr. GRPO removes length normalization and standard-deviation division, reducing biases such as answers becoming shorter once correct and advantages being amplified when difficult. The video links this to DeepSeek-R1-style reasoning training.

7. AdamW, about 7 minutes. From SGD to momentum, RMSProp, Adam; Adam entangles decay and adaptive step size, which can hurt generalization; AdamW decouples weight decay and becomes the long-term default.

8. Muon / MuonClip, about 18 minutes. Linear-layer momentum matrices are nearly low-rank, and rare directions update weakly; Muon uses Newton-Schulz etc. to orthogonalize the momentum matrix. The speaker says it is about twice as compute-efficient as AdamW—this is a claim in the video. At large scale, it is paired with decay and learning rates adjusted by matrix size; attention logit explosion uses QK-clip, and in MLA scenarios this extends to MuonClip.

9. LLM Serving, about 42 minutes. The material is marked as member content; if after watching the first 8 episodes you feel you need it, you can study it yourself.

In the comments, the author clearly says the series is still WIP and asks what topics should be added. Replies are very concentrated: some mention KV-cache, prefix reuse, and failure modes during long agent runs; some mention paged attention, because VRAM fragmentation from variable-length KV cache often bottlenecks inference earlier than compute; some suggest adding quantization and compression, including PTQ, QAT, KV cache quantization, pruning, and distillation; others mention the roofline model to explain why decode is memory-bound. Serving-side suggestions include treating KV cache as a distributed storage system, continuous batching, chunked prefill, request scheduling, and SLO-aware scheduling. On the evaluation side, people also remind about calibration, uncertainty, hallucination detection, and benchmark contamination. Data mixing ratios and cleaning are also called out; indeed, few public courses systematically cover this area.

There is an interesting detail. The author deliberately combines parallelism into one episode to compare the trade-offs of different parallelism methods; some viewers want it split into separate lectures. Marvin gave a practical way to learn: for each section, note down what part of VRAM it saves, what kind of communication it adds, and what it sacrifices, and finally string them into a comparison table. Zhou Quan's question is more direct: explaining clearly is not the same as being able to tune it—have you hand-coded it? This question is blunt, but it hits the gap between courses and real practice.

A direct judgment: the greatest value of this course lies in its order and trade-offs, not in its glossary. It explains training and inference along a single resource-constraint line, with VRAM, communication, adaptation cost, latency, RL optimizers, and serving scheduling appearing in sequence. When watching, don't just memorize names like ZeRO, GRPO, and Muon; remember what cost each of them replaces.

The course is still being updated. Those topics in the comments—data, KV cache, quantization, evaluation, and cost accounting—may be exactly the chapters most missing in the next stage.