Buoyant Open-Sources Kinference: The Kubernetes Control Plane for vLLM, Median TTFT Cut by 95%?
Buoyant has released Apache 2.0-licensed Kinference, an inference control plane purpose-built for vLLM running on Kubernetes. It delivers declarative routing, KV cache-aware load balancing and per-tenant metrics. Official benchmarks show a 95% reduction in median TTFT, but this figure comes with specific preconditions and should not be directly applied to production environments.
The bottleneck of LLM inference does not lie solely in GPUs. After vLLM is deployed on Kubernetes, a series of tricky issues come up around request routing, cache reuse, and tracking resource consumption for each tenant. Default load balancing usually operates based on connection counts or round-robin scheduling, with no awareness of the location of KV cache. As a result, requests are often sent to pods with no relevant cached data, which triggers redundant prefill work and leads to longer TTFT.
Buoyant, the company behind the Linkerd service mesh that has long specialized in control plane development, recently open-sourced Kinference, an inference control plane under Apache 2.0 license that is built specifically to solve these exact pain points. Currently designed for vLLM on Kubernetes, Kinference provides three core features: declarative routing policies, KV cache-aware load balancing, and per-tenant metrics.
Declarative routing policies mean you can use configurations to define exactly how requests should be routed, for example, diverting traffic based on model type, tenant identity, or priority level. KV cache-aware load balancing is the key highlight: it routes requests directly to pods that already hold the relevant KV cache to cut down redundant computation. Per-tenant metrics make it possible to fully observe usage and latency metrics under multi-tenant scenarios.
In Buoyant's own benchmark tests, the median TTFT is reduced by 95%. This eye-catching number comes with important fine print: it is sourced from Buoyant's internal benchmarks, not independent third-party testing. Full details of the model, workload, hardware, and the baseline used for comparison are not laid out in the brief release announcement. TTFT is extremely sensitive to cache hit rates: performance improvements will be dramatic if the workload has obvious shared prefixes, while the effect could be negligible if incoming requests have no correlation with each other at all.
So who is Kinference suitable for? Teams that are already running vLLM on Kubernetes and running into routing and cache reuse issues. It is especially ideal for scenarios with multi-tenant workloads, shared system prompts, and strict latency requirements for time to first token. It is not a magic plug-and-play component that boosts performance instantly, as the control plane itself has deployment and configuration overhead.
Project repository and original article.
发布时间: 2026-11-07 16:16