49 LLM Inference Interview Questions Open-Sourced, Backed by a Hands-On Roadmap of 48 Topics
TensorTonic has compiled high-frequency Inference/LLM engineer interview questions into an open-source handbook: 49 questions, each with an explanation and an exercise you can run tests against, organized into seven stages covering everything from prefill/decode to production serving. Memorizing answers is not enough—writing quantizers and schedulers by hand is the real focus.
TensorTonic has open-sourced a repository called the LLM Optimization Handbook. It collects 49 questions that appear frequently in Inference/LLM engineer interviews, each with an explanation and a practice exercise you can run against tests.
The repository is here. It connects 48 topics across seven stages, from how a single request runs through the GPU all the way to how a model is served in production. Each topic links to a short explanation and a practice exercise implemented in Python, CUDA, or Triton and run against tests.
The repository states that most optimization advice is easy to repeat but hard to apply, and that writing your own quantizer or scheduler is the fastest way to see what each technique actually changes. That sentence highlights how it differs from an ordinary question bank: the question bank is just the entry point; the roadmap and exercises are the main body.
The seven stages roughly cover:
- Stage 1, Inference Fundamentals. How requests are served, attention forward pass, prefill and decode, roofline, memory budget and GPU count, token sampling, TTFT, TPOT, ITL, throughput, latency percentiles.
- Stage 2, Quantization. Numeric formats and mixed precision, symmetric INT8, per-channel scale, group-wise INT4, FP8 block quantization, outliers and GPTQ/AWQ, QLoRA.
- Stage 3, KV Cache and Attention. KV cache size estimation, MQA, GQA, MLA, RoPE during decoding, FlashAttention and online softmax, PagedAttention, prefix caching.
- Stage 4, Speculative Decoding and MoE. draft and verify, acceptance rate and real speedup, draft-free variants such as Medusa and EAGLE, top-k expert routing, token dispatch and aggregation.
- Stage 5, GPU Kernels. threads, blocks, warps, memory coalescing and shared memory, tiled matmul, reduction and softmax, Triton kernels, kernel fusion, attention kernels.
- Stage 6, Batching and Parallelism. dynamic batching, continuous batching, chunked prefill, prefill/decode disaggregation, cache-aware routing, collective communication, tensor parallelism, expert parallelism.
- Stage 7, Serving in Production. Building an inference server, benchmarking methods, GPU-time profiling, throughput and cost under latency SLOs, rollout and monitoring.
In interviews, people often treat quantization, FlashAttention, and continuous batching as three universal trump cards. Once the follow-up questions ask how to estimate KV cache memory, how to handle INT4 outliers, or how scheduling strategy affects TTFT, those cards are no longer enough. The value of this handbook is that it breaks these terms down into measurable, reproducible exercises.
The repository also offers three usage paths. For ML engineers moving into inference, go through the stages in order and do at least one question per topic. For preparing for inference or ML systems interviews, the first three stages plus the batching stage cover most questions. For those already serving models in production, use the table as a checklist and focus on filling gaps in topics you have only used through library flags.
Prerequisites are not high: familiarity with Python and a rough understanding of how transformers work. No GPU programming experience is needed before Stage 5. The repository uses the MIT license.
If you just want to grind questions, start with the first three stages and batching. If you are already tuning online services, use the seven-stage table as a checklist and fill in the parts you have always left to frameworks and never written by hand.
发布时间: 2026-11-19 01:32