Wink Pings

Want to run your own inference service? Inference Stack installs the whole stack with one command

Inference Stack is an open-source integrated solution released by the LiteLLM team. It packages vLLM, llm-d routing, the LiteLLM API gateway, and Grafana monitoring into a deployable stack that starts with one command and is operated through a web UI. It can also run in simulation mode without a GPU to experience the full workflow.

LiteLLM is a large language model API gateway project, and its team BerriAI has released Inference Stack: an open-source solution that packages the components needed for inference services. To run your own LLM application, beyond the model itself you also have to handle infrastructure such as deployment, routing, and monitoring. Inference Stack compresses these things into a single command.

This stack contains five parts:

ComponentRole
Hugging FaceModel source; search for models and view GPU requirements in the deployment UI
vLLMInference engine; runs models on your GPU
llm-dRouting layer; selects vLLM replicas based on cache and queue data
LiteLLMUnified API entry point; manages all models and supports chat testing
GrafanaMonitoring dashboard; shows model health, request traffic, and response times, with data from Prometheus

Among the five components, llm-d is the most noteworthy. It performs deployment-level routing: based on the cache hit status and queue length of each vLLM replica, it distributes requests to the appropriate replica. Cache hits can skip duplicate computation, directly affecting inference cost and response latency. In multi-replica scenarios, this layer is directly tied to GPU utilization.

Inference Stack architecture flow: LiteLLM routes requests, llm-d selects vLLM replicas, Hugging Face provides weights, Grafana displays metrics

There are three deployment options: Docker Compose, Kubernetes, and AWS. For a Docker quick start, just run:

curl -fsSL https://github.com/BerriAI/inference-stack/releases/download/v0.3.0/install.sh | bash -s -- docker

After running it, open http://localhost:8080 in your browser and log in with admin and the generated password. Go to Deployments, select a model, and click Deploy. Once the model is ready, click Try in LiteLLM to send a message and test it directly.

Real model inference requires an NVIDIA GPU. Machines without a GPU can enable simulation mode in the model settings to return preset responses; the full workflow still runs through as usual.

Screenshots of several core interfaces:

vLLM launcher: search models from Hugging Face and estimate GPU requirements

LiteLLM model hub: a unified entry point for serving models through vLLM

llm-d routing interface: two simulated replicas, with configurable cache and queue settings

Grafana dashboard: model health and request traffic for a simulated deployment

Someone in the announcement thread asked: why not use SGLang? The vLLM ecosystem is more mature, and integration with Hugging Face and LiteLLM costs less. SGLang has performance advantages in some scenarios, but as the default inference engine, vLLM is the more stable choice.

Inference Stack uses the MIT license and is currently a v0.3.0 preview. The code is on GitHub, and the documentation is at inferencestack.io.

If you are evaluating self-hosted inference infrastructure, this stack is worth running through once. It packages models, routing, APIs, and monitoring together, compressing the engineering glue work into a single command. All that is left to solve is where the GPU comes from.

发布时间: 2026-11-23 03:21