Don't Rush into CUDA: This 56-Chapter Open-Source GPU Fundamentals Note Set Is Free

The most common mistake in GPU tutorials is teaching the API first and then having people memorize performance parameters. Getting a kernel to run isn't hard; knowing why it's slow is.

Tech with Mak recommended a free open-source resource in a post called Before GPU Engineering. The repository is here, and the online digital notes are here. It explains GPU architecture and performance engineering from the hardware level upward. Currently 56 chapters are online, covering Phases 1 through 4, and Phase 5 is still being written.

The repository author states that these notes build from logic gates and memory buses all the way up to GPU architecture, without skipping over things with abstract terminology. The table of contents includes CPU and GPU architecture, parallel computing, warps, SIMT, thread blocks, registers, shared memory, caches, HBM, compute-bound vs. memory-bound, kernel launch, synchronization, races, memory coalescing, occupancy, latency hiding, register spilling, arithmetic intensity, tiling, GEMM, and profiling.

Its ordering is worth noting. Many materials jump straight into CUDA or Triton, first getting code to run and then circling back to fill in the hardware. These notes do the opposite: they first explain how the GPU executes programs, moves data, and handles parallel workloads, and only then discuss kernel optimization. Reading in order may seem slow, but it actually saves time later on blind parameter tuning.

The Phase 2 table of contents focuses on the translation layer, hardware hierarchy, synchronization and races, scale, and performance. For example: why C++ doesn't automatically compile to the GPU, why a kernel isn't an ordinary function, why threads are grouped, why shared memory is fast, kernel launch overhead, coalesced vs. strided access, and why GPU programs become slow.

Phase 3 leans more toward performance engineering. The memory wall, arithmetic intensity, dual ceilings, register physics, what the occupancy ratio means, register spilling, global memory latency, memory coalescing, DRAM row-buffer thrashing, predication, compare-and-swap, register tiling, and finally GEMM and profiling. One reader asked which phase covers memory coalescing; the answer is in Phase 3's Global Memory & Coalescing, with Chapter 44 specifically covering the difference between 32 threads in one transaction and 32 transactions.

Who is it for? The repository author targets developers, students, and self-learners who know basic programming but feel GPU is a black box. For AI/ML engineers, it can fill in the hardware intuition behind LLM training and inference. For general readers, these terms may look intimidating; reading them slowly as if they were a hardware manual is less stressful than grinding through CUDA syntax directly.

One reader already knows PyTorch and asked which phase to start from. The repository author's answer is direct: read sequentially from Chapter 1. The concepts depend on one another layer by layer; skipping shared memory makes matrix tiling hard to understand. Another reader asked about prerequisites and projects: a basic understanding of programming is enough to start reading, but if you want hands-on practice you'll need to supplement CUDA or Triton yourself. Tech with Mak also notes that this is primarily a reading resource, not heavily weighted toward hands-on programming courses, and some later chapters are still being written.

One reader said they like how it discusses GPU architecture and performance together, and that many bottlenecks ultimately come back to the hardware. That judgment is more useful than memorizing APIs.