Hand-writing a mini vLLM on Mac: tiny-llm breaks inference optimization into four weekly assignments

There is an open-source course on GitHub called tiny-llm. It asks you to build a mini vLLM-style inference engine by hand using MLX, C++, and Metal on Apple Silicon, centered on Qwen3.

Tech with Mak introduced it in a tweet, and someone in the comments noted that the project author is @iskyzh. The repository already has over 4,800 stars. The course is divided into four weeks: Week 1 starts with matrix operations and writes attention, RoPE, GQA, RMSNorm, MLP, sampling, and the autoregressive loop, first producing a model that can generate text. Week 2 begins performance work: add KV cache, retain packed 4-bit weights, write SIMD matrix operations, then work on fused RMSNorm, RoPE, SwiGLU, and tiled prefill attention. Week 3 moves into the serving layer: continuous batching, chunked prefill, paged KV, and attention that reads directly from non-contiguous cache pages. Week 4 builds a small coding agent harness, including validated tool calls, checkpoints, context management, and evaluation.

The value of this course lies in its path. It breaks down the hardest parts of inference engineering into checkpoints you have to fix yourself: each step has exercises, reference implementations, tests, and measurements. What the course gives you is how memory traffic, kernel occupancy, KV cache growth, batching, and request scheduling pull against one another—not a conclusion about how much faster some optimized code became. A commenter in the replies said hand-writing KV cache was the lesson easiest to remember: decode is memory-bandwidth-bound, speed comes from not recomputing, and then the cache itself becomes something that eats memory. This observation is more useful than memorizing concepts.

The environment requirements are also worth spelling out. The course is based on MLX and Metal on Apple Silicon and does not require an NVIDIA GPU. The introductory path supports running Qwen3-0.6B on an 8GB Mac. The repository notes mention Qwen3-4B, used to expose real weight bandwidth, attention, and cache costs while still allowing local iteration. Windows is not the target environment for this course. If you want to follow along, first do the environment setup in the book, then run the installation check provided in the repository.

One caveat: some later checkpoints are still under development and review. This is not a crash course after which you can declare that you have replicated vLLM. It is more like a construction blueprint backed by tests and measurements: you install the parts of an inference engine week by week, and you investigate for yourself where it is slow, where it is wrong, and where it should be quantized.

Repository: https://github.com/skyzh/tiny-llm
Course book: https://skyzh.github.io/tiny-llm/