KubeCon Inference Six-Layer Stack: DRA Is Stable, but the Scheduler Still Doesn't Preempt Resources
Michael Levan breaks down the inference infrastructure at KubeCon into six layers. GPU, vLLM, gateway, llm-d, sandbox, and Kubernetes are all in the diagram. In the replies, a limitation that production teams should pay more attention to was added: DRA is stable, but the scheduler does not preempt DRA resources.
KubeCon's theme this week is inference. The term is broad—broad enough to encompass GPU allocation, model serving, gateway routing, and sandbox isolation. Michael Levan breaks these discussions down into six layers; the original post includes a diagram of inference infrastructure and agent infrastructure.
These six layers come from his breakdown; Kubernetes does not officially have such a layering. What is useful for production teams is that each layer corresponds to a concrete problem.
- GPU and memory. Whether there is enough is one layer; how to allocate and where to share it is another. DRA is mentioned multiple times and handles GPU slicing; KEDA and Karpenter manage scaling.
- vLLM serves LLMs and uses PagedAttention for KV caching. The goal is to waste less VRAM and improve throughput.
- Inference-aware gateway. An AI Gateway with the Gateway API Inference extension, such as @agentgateway, is responsible for routing requests to LLMs.
- Distributed inference and scheduling. llm-d performs cache-aware request scheduling across vLLM instances.
- Isolation. Agents run on this infrastructure and need real sandboxes, with both ingress and egress controlled. The sandbox layer must come from the underlying engine and balance isolation and scale. @kagent_dev and Agent Substrate are mentioned.
- Where it runs. Kubernetes.
Together, these six layers form the inference infrastructure and agent infrastructure for serving LLMs and agents. A nice diagram does not make production problems disappear on its own.
Amitendra Singh added a key limitation in a reply: according to the Kubernetes documentation, DRA has been stable since v1.35, but the scheduler does not preempt DRA resources. A high-priority Pod waiting for an already-claimed device will remain Pending until the Pod holding the device finishes or is deleted. GPU slicing and dynamic allocation sound flexible, but with preemption semantics not fully addressed, capacity planning cannot look only at utilization.
Macci X's reply is sharper: calling everything inference effectively means latency is no longer owned by any specific person. Another reply simply said Engineering.
The six-layer stack can serve as a troubleshooting map. The truly hard engineering part is being able to clearly state, for each layer, who owns latency, who owns capacity, and who owns isolation. Otherwise, inference is just a new name, and Pods that should be Pending will still be Pending.
Original thread: https://x.com/TheNJDevOpsGuy/status/2121736044232274365
发布时间: 2026-11-15 18:32