Kubernetes Can Handle vLLM, but It Cannot Handle a Million Sandboxes

Karthik asked a not-complicated but easy-to-argue-about question on X: Is Kubernetes the right place to run AI inference and agent workloads, or are we forcing a new kind of workload into infrastructure that was not designed for it?

Niels Rogge's response was blunt: yes, vLLM's production stack runs on Kubernetes. vLLM and SGLang are currently two commonly used frameworks for deploying LLM inference.

vLLM's production stack is a Kubernetes-native, cluster-level deployment solution. The capabilities listed in the official documentation include: scaling from a single instance to a distributed deployment without changing application code, a web dashboard for metrics, performance gains from request routing and KV cache offloading, and one-click deployment to AWS, GCP, or other clouds. Among the specific features used, KV Cache Aware Routing, Load Aware Routing, Prefix Aware Routing, Disaggregated Prefill, cross-instance shared KV Cache, KEDA autoscaling, and semantic routing are all listed in the documentation's use cases.

So on the inference side, Kubernetes is not merely usable with some effort; there is already a whole set of engineering practices that has grown up around it.

But Niels added a caveat in the same tweet. If the discussion is about sandboxes—the kind of environment agents use to execute code and operate computer screens—Modal wrote an article explaining why Kubernetes is not the optimal solution for this workload, and how they scaled sandboxes to more than 1 million concurrent instances.

Modal's article puts it bluntly: running one million sandboxes will push any container platform to its limits. The reason is not only the number of containers, but also the tens of thousands of compute nodes required behind them. Many operations have complexity O(containers), O(nodes), or both, and traditional container platforms hit a wall at this point.

They use Kubernetes as an example. In the worst case, K8s's scheduling algorithm is O(n x p), where n is the number of nodes and p is the number of pods, and scheduling is serial by default. Each pod writes to etcd multiple times during its lifecycle. This centralized persistent store will run into serious problems under high pod creation rates or high churn, and etcd cannot natively shard at the keyspace level. There is also a more fundamental point: each node writes to etcd at least once per heartbeat cycle to report liveness, so the baseline write load alone is O(nodes), completely independent of pod creation.

Kubernetes can certainly scale, but the cost is not small. To run large numbers of nodes, etcd usually has to be rewritten or replaced. To increase scheduling throughput, you have to build a complex scatter-gather system to parallelize the scheduling algorithm while still maintaining a single source of truth for pod state. Sharding and parallelism are not things you get by default, because strong consistency is the backbone of K8s's design.

Modal's own old architecture had similar problems. They also relied on strong consistency; creating and scheduling sandboxes required global coordination, writes to Postgres were O(sandboxes), and it could not be easily sharded.

So they simply rewrote it. The core idea of the new architecture is: any workload that is O(sandboxes) or O(nodes) must be horizontally scalable by default; the creation path should be as simple as possible; all central coordination should be removed, trading global consistency for scalability and performance.

Specifically, there is no longer a single serial scheduler. Instead, a group of scheduling servers handles creation requests concurrently, and each server runs a fast scheduling algorithm using data cached in memory. Each worker is itself the source of truth for its state and periodically publishes state to a Redis stream; scheduling servers asynchronously consume this state to make decisions. Once a decision is made, the scheduling server contacts the worker directly via RPC to request sandbox creation. The worker accepts if it has idle resources and rejects if it does not. There is no data store anywhere in the creation path; metadata and results are mostly persisted asynchronously.

The result is that the creation path needs only two network hops and one cheap CPU operation. There is no central bottleneck and no single point of failure, so there is no practical ceiling on aggregate scale or creation throughput. In their load test, they ran one million sandboxes; all were created in under a minute, and the median startup time was under half a second. Along the way they also hit a kernel pitfall: the new scheduler pushed containers to workers so fast that when large numbers of containers started at the same time, they contended on Linux's rtnl lock, container network rules were created extremely slowly, and in the end they had to change the sandbox's container network configuration.

Looking back at the replies under that tweet, they already make this split clear. One commenter summarized it in a sentence: One AI label, two workloads. Anika said inference is fine, agents maybe. Lakshya complained that GPU pod scheduling really hurts. Haroon mentioned Kubeflow, saying K8s provides the foundation, but AI workloads need more than bare K8s. Avinash was more specific: K8s is a good foundation, but it needs additional layers for efficient LLM inference; KV cache aware routing and prefill/decode disaggregation are exactly what projects like Kthena and llm-d are solving. Karthik himself was also looking at kserve and kthena, preparing to read the code and contribute.

These projects are all layers added around Kubernetes. This precisely illustrates one thing: bare K8s is not enough. People are not abandoning it; they are adding on top of it the parts that AI workloads need. On the inference side, what they add is routing, caching, and splitting prefill and decode. On the sandbox side, the problem is more fundamental; adding layers cannot solve it, and Modal simply changed the foundation.

So the question itself can be broken apart. AI is a bucket; once inference and sandboxes are put into it, their workload characteristics are almost opposite. Inference is a long-running service: instances stay alive, state matters, and traffic predictability is relatively high. Sandboxes are short-lived, high-churn, enormous in number, low-value-per-instance disposable containers, and peak creation rate is the real crux. Kubernetes is good at the former, because its strong consistency, rolling updates, and service discovery were designed for long-running services. The bottleneck for the latter is precisely that strong-consistency model; every etcd write is stepping on the brakes for bursty creation.

If you insist that K8s is unsuitable for AI, half of the inference practitioners will disagree. If you insist that K8s is suitable for all AI workloads, Modal's one million sandboxes are a counterexample. Where the debate gets stuck is not in the technology, but in the label.