How Exactly Do LLM Servers Handle High Concurrency? This 41-Minute Video Breaks Down vLLM and SGLang

Jia-Bin Huang has released a 41-minute video with a very direct title: How LLM Serving Actually Works. It answers a concrete question: when you type a sentence into a chat box, what does the server actually do; and why, when serving many people at once, latency and throughput pull against each other.

The full video is on YouTube: https://youtu.be/T4SRu-trimA

The video does not stop at the conceptual level. It uses vLLM and SGLang as working examples and breaks things down along the path a request takes after entering the server: scheduler, KV cache, model, sampler. The chapters start with next-token prediction and go on to cover PagedAttention, prefix caching, structured output, speculative decoding, FlashAttention, CUDA graphs, quantization, tensor parallelism, and MoE expert parallelism.

For ordinary users, two facts are most worth understanding first. First, generating every token in an LLM requires repeating a large amount of computation, so the server caches the Key and Value it has already computed; this is the KV cache. Second, a request has two phases: prefill processes the input, and decode outputs tokens one by one. Prefill is more compute-bound, while decode is more memory-bandwidth-bound. Many optimizations cannot avoid this difference.

The scheduler's job is to pack different requests into batches to improve throughput. continuous batching (Orca) lets new requests avoid waiting for the whole batch to finish, and the scheduler also has to handle priorities; chunked prefill (Sarathi-Serve) splits long inputs into chunks, and goodput is about how many requests are completed within a latency target. The video also mentions putting prefill and decode into different pools, which is a trade-off between throughput and latency. For developers, there is a common misconception here: larger batches mean higher throughput, but per-request latency may get worse. The scheduling policy determines which metric you are optimizing.

KV cache consumes GPU memory. The video uses an analogy to OS virtual memory and explains how PagedAttention divides the cache into blocks; when the GPU memory pool is full, it triggers preemption. The video also separately discusses how much memory KV cache requires. Shared prefixes are another key point: system prompts, chat history, and agent context are often repeated. vLLM uses hashed blocks for prefix caching, while SGLang uses RadixAttention to maintain a radix tree of cached prefixes. When cache is insufficient, there are also eviction and HiCache. Cache-aware scheduling prioritizes matching the longest prefix, but the video also points out the cost: fairness is affected. A high cache hit rate does not mean every request gets faster; the queueing order changes.

The structured output section is very practical for application development. JSON, regex, and state machines constrain decoding, and XGrammar builds compressed state machines. It explains that constrained generation happens during the decoding phase and cannot be made up for by post-processing. Speculative decoding, by contrast, uses a small model or Medusa, EAGLE, or n-gram to draft first and then verify. It trades cheap drafts for fewer decode steps, provided verification is not too expensive.

The low-level optimization section is more about squeezing fixed overheads: FlashAttention, kernel launch, CUDA graphs, torch.compile, overlap scheduling, as well as FP8 KV cache and quantization. The large-model deployment section discusses DeepSeek-V3 on 8 GPUs with tensor parallelism, latent attention, data-parallel attention, and MoE wide expert parallelism. Finally, it lands on replicas and a cache-aware router. The video closes with three principles: never idle, never wasteful, never twice. In plain terms: don't let the GPU sit idle, don't waste GPU memory, and don't recompute the same thing.

These three phrases are more useful than many architecture diagrams. There is no single silver bullet for LLM serving optimization; scheduling, caching, parallelism, and quantization each solve a piece of the constraints. The video's chapters are long. If you only want the main thread, you can start with 0:00 to 16:40, where scheduling, KV cache, PagedAttention, and prefix caching are all covered; afterward, jump to structured output, speculative decoding, and parallelism according to your interests. For AI practitioners, this chapter list can serve as an index for troubleshooting performance problems; for ordinary users, it can at least clearly explain why the first token is slow, why long context is expensive, and why caching system prompts is meaningful.