Wink Pings

Finish These 12 Inference Infrastructure Projects and You're Hired

Suraj Sharma posted a list of 12 projects on X and said that if you can build them, you'll get hired. It covers Triton kernels, KV cache, MoE, quantization, speculative decoding, semantic caching, and edge fallback, with data verified by other users.

Suraj Sharma posted a list on X: 12 inference infrastructure projects. If you can build them, you're hired.

The list is very concrete—not the 'build a chatbot' kind. Each item targets a real bottleneck in the inference pipeline. This industry doesn't lack people who watch tutorials; it lacks people who can bring down latency and cost while pushing throughput.

Group 1: Understand the hardware, not just call APIs

  • Write a fused attention or RMSNorm kernel in Triton, profile memory bandwidth with Nsight Compute, and compare it against native PyTorch. This shows that you understand roofline analysis instead of treating the GPU as a black box.
  • Calibrate a frontier model to FP4/INT4, measure the exact perplexity degradation, and quantify the latency and memory gains. Quantization isn't hard; what's hard is making the quality loss clear.
  • Run a local 2B draft model to generate candidate tokens for a 70B target model, then verify them in a single forward pass. Speculative decoding with zero quality loss and a 2x decoding speedup.

Someone pointed out that memory access patterns deserve more attention than computation itself: profile with perf first, then apply CUDA tricks.

Group 2: Scaling Serving, with Memory at the Core

  • Hierarchical KV cache offloading: GPU→CPU→SSD, without blocking the decode loop. This determines whether you can serve ultra-long contexts without going bankrupt.
  • MoE expert-parallel router: shard expert weights across multiple GPUs, optimize all-to-all communication, and balance expert load. Trillion-parameter sparse models can't avoid this.
  • Disaggregated prefill/decode clusters: prefill is compute-bound, decode is memory-bound. Run them on different hardware connected via RDMA/InfiniBand to maximize MFU.

Group 3: Cost and Reliability Are Engineering Problems

  • Implement the sampling loop with a context-free grammar to enforce valid JSON/Pydantic output at the token level. Don't rely on prompting and luck.
  • Semantic cache: if embedding similarity is above 98%, return the cached response directly without touching the GPU. But someone pointed out that semantically similar prompts may need different answers, and a false hit is a correctness problem.
  • Complexity routing: based on prompt entropy, route simple questions to a quantized 8B model and complex reasoning to a 405B API. Cost drops 60% without degrading user experience.
  • WebGPU/MLX edge fallback: when the cloud API returns 503, degrade to a 3B model running in the browser or on Apple Silicon.
  • Queue-based GPU autoscaling: KEDA scales spot instances based on the depth of the pending-token queue and gracefully drains them when idle. Idle GPUs are just burning money.

Group 4: Engineering Judgment

  • Publish an inference benchmark teardown, plot Pareto curves for cost vs TTFT vs throughput, and compare vLLM, SGLang, and TensorRT-LLM. You need both scientific rigor and the ability to explain the conclusions clearly.

Below the list, other users added real data:

KV cache memory per token is roughly 2 × layers × KV heads × head dim × bytes. Llama 3.1 8B under bf16 is about 128KiB per token, so an 8k context is 1GiB. GQA, paging, and KV quantization all exist to deal with this.

Another user verified speculative decoding:

The same 8-bit Qwen with its built-in multi-token prediction head went from 71–77 tok/s to 108–123 tok/s, without changing the model.

Some users also said that learning batching first is more useful for understanding real serving costs than a single kernel. Others said tech stacks go out of date, but foundational systems knowledge compounds. Someone poured cold water on it: no one is hiring right now. The most sarcastic comment appeared at the end:

Project 13: Explain to finance why the GPU bill tripled after 'optimization.'

Suraj also posted a 12-stage learning path, from C/C++ and CUDA to AI Gateway and edge inference. But the project list itself already makes the point: the barrier to inference infrastructure is understanding hardware, memory, and cost—not just getting a demo to run. The role of Rust in AI infrastructure also deserves a separate discussion.

Suraj added a line himself: one completed project with real benchmarks beats twelve unfinished experiments.

发布时间: 2026-11-22 06:13