Was “Inference Engineer” a job title invented only last Tuesday?

There’s a post on X listing 6 resources that teach people how to become an AI inference engineer in 2026. The author says don’t just focus on training models; you need to learn how to make models run faster, cheaper, and able to handle scale.

The top-voted reply was from X user Jacob Steele: “Inference engineer sounds like a job title invented last Tuesday.” In other words, an inference engineer sounds like a position invented just last Tuesday.

This comment stings, but it’s half right.

The job title is indeed new. Search job boards and you’ll find that a large number of inference-related job descriptions have only spread out over the past year. But it’s wrong about the other half: the work itself is not new. Since the day the first model went live, people have been doing this math.

What problem does inference engineering solve? Hugging Face’s documentation gives a set of numbers: a 70B-parameter Llama 2 requires 256GB of memory for full-precision weights, and 128GB even at half precision. The strongest A100 and H100 at the time had only 80GB of VRAM. You couldn’t fit one model in, let alone serve multiple users at the same time.

Training is how you build the model; inference is how you make the model run in real-world scenarios. The voice assistant on your phone, the automatic replies in customer service systems, and the completions in your code editor all rely on inference. Training is a matter for a few companies; inference is something every company using AI has to face.

That post listed 6 resources, covering the path from getting started to deployment.

1. Inference Engineering (inferenceengineering.tech)

An online book by Philip Kiely, published by Baseten, 2026 edition. 8 chapters, covering everything from GPU hardware and CUDA kernels to quantization, speculative decoding, and production deployment. It has animated diagrams, a VRAM calculator, a KV cache capacity calculator, and more than 100 practice questions. Suitable for systematically going through the entire inference stack.

2. Hugging Face’s LLM inference optimization documentation

More hands-on. Static KV cache with torch.compile can deliver up to 4x speedup; speculative decoding uses a small model to draft for a large model; FlashAttention-2’s tiled computation reduces VRAM reads and writes; quantization compresses weights to 8-bit or even 4-bit. Each technique comes with code examples. Suitable for people who have already written model-calling code and want to dig one layer deeper.

3. vLLM documentation

A high-throughput LLM serving framework. Its core is PagedAttention, which manages KV cache in pages much like an operating system manages memory, reducing fragmentation. If you want to deploy models yourself, it’s basically unavoidable.

4. SGLang documentation

Positioned similarly to vLLM, with a focus on structured generation and prefix caching. If your scenario has a lot of repeated system prompts, or the model output must strictly follow JSON format, it’s worth a look.

5. NVIDIA TensorRT-LLM

NVIDIA’s own inference stack. It covers compilation optimization, quantization, and multi-GPU parallelism. Its performance ceiling is high, at the cost of locking you into the NVIDIA ecosystem.

6. GPU MODE

A community on GitHub that teaches GPU programming, CUDA, and Triton. If you want to figure out what’s actually being computed at the kernel level, start here.

The post ends by saying the goal is not just to get the model running, but to serve millions of tokens with lower latency, higher throughput, and minimal GPU cost.

Broken down, it’s exactly the things above: how to manage KV cache, how to assemble batches, how many bits to quantize weights to, what implementation to use for attention, and how to schedule requests.

Some say this is a job title invented last Tuesday. That’s also true. The name is new. The problem is old.

Since the day the first model went live, people have been doing this math.