Memorizing These 25 Inference Interview Questions Still Won’t Get You Through—the Comments Tell the Truth
An AI inference interview list being passed around on X, with 25 questions ranging from KV cache calculations to million-DAU architecture. But what’s really interesting is the thread of replies underneath: someone takes the KV cache math down to the decimal, someone points out that if GPU utilization won’t go up, don’t start with Nsight, and someone says outright that question 3 is the real filter.
A 25-question list is making the rounds on X, with a very direct title: if you only have 24 hours to prepare for an AI inference engineering interview, work through these in order.
The post is by Vishal Singh. The list starts with “what happens between a request coming in and the first token going out” and runs all the way to how to estimate peak QPS, GPU capacity, and cost when designing an inference system with a million daily active users. In between are how to calculate KV cache, what problem PagedAttention solves, why FlashAttention is faster even though the math is unchanged, how continuous batching schedules under latency deadlines, when speculative decoding hurts more than it helps, and how to choose between FP8 and INT4.
The 25 questions cover a complete inference pipeline: request path, memory management, attention mechanisms, batching and scheduling, quantization and hardware selection, performance profiling, observability, multi-tenancy, MoE parallelism, and cost optimization.
But the list itself is not the point. The comments are.
Someone asked which of the 25 is most likely to stump people. Vishal’s own answer was that the metrics-heavy ones do the most damage; if you’re not solid on the first 19, questions 20 to 25 are basically unanswerable. Ayaan Khan added that question 3 is the real filter. KV cache sizing is where textbook knowledge meets a spreadsheet, and most candidates have never calculated it by hand.
Harrison just did the math. Assume 80 layers, 8 KV heads, 128 dimensions per head, and FP16 precision: 2×80×8×128×2×32768, so a full 32K context is about 10 GiB. With 100 users who don’t share context, KV cache alone is 1,000 GiB, before counting model weights and runtime overhead. Once you put that number on the table, it’s obvious why question 3 filters people out.
On question 5, Vanraj Narula gave the one-sentence version: FlashAttention does the same math, but it tiles the computation into on-chip SRAM, so the full attention matrix is never written to HBM and read back from start to finish. What it wins is IO, not FLOPs.
Under question 14, Sunil Mehta left a very specific troubleshooting path. If a GPU only reaches 30% utilization under load, it’s usually not a kernel problem—look at the scheduler first. max_num_seqs or the KV cache block budget is throttling the batch; check KV cache usage and queue depth, then decide whether to open Nsight. He also gave a number for question 20: estimate warm-up headroom as cold-start time multiplied by traffic ramp speed; pulling a 140 GB checkpoint plus CUDA graph capture takes several minutes, so the weights need to be staged on local NVMe in advance.
Someone asked whether undergraduates can get inference roles. Vishal’s reply was short: at this point in time, anyone can get anything; skills matter more than everything else.
Someone else asked how anyone could get through 25 questions in 24 hours. He replied with a laughing-crying emoji.
Jay shared his experience writing a Triton kernel: layer norm hit 1 TB/s on an A100 and beat PyTorch at large sizes.
At the end of the original post there is a reminder: don’t memorize answers. Understand the concepts, calculate them with real numbers, and focus on the trade-offs. The people in the comments who calculate KV cache down to GiB, break FlashAttention into an IO problem, and point to the scheduler for 30% utilization are doing exactly that.
发布时间: 2026-10-25 01:40