What the LLM Course Is Missing: The Comments Section Pushes KV Cache and Scheduling to the Front

Jia-Bin Huang updated an LLM training and inference playlist/course on X, with the status marked WIP. The topics already covered include distributed training, fine-tuning, RL, OPD, serving, spec. decoding, and basic optimizers. He posted asking what else should be added. The original post is here, and the playlist is here.

The comments section quickly turned into minutes from a free-course committee. Some put KV cache and prefix reuse first and also warned about failure modes during long agent runs. The suggestion is very practical: once long tasks run late, context caching, prefix reuse, timeouts, and state management are more likely to trip people up than another chapter on optimizers. Jia-Bin Huang replied that a video on prefix reuse is coming soon and continued to ask for references on long-agent failure modes.

Another group of suggestions focused on inference serving. One commenter named paged attention because variable-length KV caches fragment GPU memory, and memory fragmentation often kills inference earlier than compute does. Another discussed KV cache as a distributed storage system and listed continuous batching, chunked prefill, request scheduling, and SLO-aware scheduling. Another commenter added the roofline model to explain why the decode phase is limited by memory bandwidth and why quantization and compression can help. The quantization part includes PTQ and QAT, KV cache quantization, pruning, and distillation. Jia-Bin Huang replied that the serving video is coming soon and that he will also try to incorporate quantization.

Parallel training was also mentioned. Some hoped all parallelization methods would be split into separate videos. Jia-Bin Huang explained that he deliberately put parallelism in one video so that the motivations and trade-offs of different parallelization types could be compared. This response is worth noting: in course design, splitting things up and putting them together for comparison each have costs; the comments section wants convenience for reference, while the author wants to build a sense of trade-offs.

Evaluation and trustworthy AI were singled out separately. A commenter suggested adding calibration, uncertainty estimation, and hallucination detection, and added a rather pointed observation: post-training often makes models more confident without making them more correct. Further on came evaluation and contamination—that is, benchmark numbers can be misleading. Jia-Bin Huang replied directly that this direction is very interesting but that he has zero knowledge about it and asked for references. This gap is more worth filling than adding another hot topic.

There were also scattered but useful suggestions: how to train trillion-parameter multimodal models; RLHF; the evolution of attention, including FlashAttention, MLA, residual, etc.; how to build datasets, with someone mentioning Hugging Face's FineWeb guide; harness design, autoresearch, RSI; and using one long video to explain common RL methods for LLMs. Someone also reminded him to keep the style of mathematical explanation, saying that was the most fun part. Another commenter shared similar content they had made, with the link here.

Looking at all these replies, what is truly scarce in the course list is failure modes in production environments. There are already many knowledge points on training and inference; KV cache management, request scheduling, quantization trade-offs, evaluation contamination, and long agents crashing are closer to real-world pitfall scenarios. The optimizer chapter can continue to exist, but the comments section has already made the priorities very clear. The next step is to see how the author fits these suggestions into the playlist and whether he will actually add the trustworthy AI part.