What is llm-d?

llm-d is an open-source project that runs large language model serving across an entire Kubernetes cluster — an orchestration layer that turns a fleet of vLLM servers into one fast, reliable inference service.

Here's the problem it solves. A single serving engine handles one machine well, but real production traffic — and the largest open models — need many machines working together. Naive load balancing wastes work, because LLM requests aren't interchangeable: some servers already hold a conversation's cached context, and different phases of a request want different hardware.

llm-d applies a few tricks at cluster scale:

• Smart routing. Requests go to the server that already has the relevant context cached, not to a random one.
• Prefill/decode disaggregation. Reading your prompt and generating the reply are different jobs, so llm-d can run them on separate, specialized groups of machines.
• Cache management. Overflow context can spill to CPU memory or disk instead of being thrown away and recomputed.

The project launched in 2025, led by Red Hat with founding contributors including Google Cloud, CoreWeave, IBM Research, and NVIDIA. As of July 2026 it's a Cloud Native Computing Foundation sandbox project, and Red Hat ships it as part of its commercial AI platform.

Be honest with yourself before adopting it: llm-d only makes sense if you're operating serious multi-GPU infrastructure. For what's happening one level down, see LLM inference.