What is vLLM?

vLLM is an open-source engine for serving large language models at scale — the software companies run on GPU servers so one model can handle thousands of simultaneous users without falling over.

It came out of UC Berkeley's Sky Computing Lab in 2023 and is now hosted by the PyTorch Foundation. Its signature idea is PagedAttention: instead of reserving one big block of GPU memory per conversation, it slices the model's short-term memory (the KV cache) into small pages and hands them out on demand — the same trick operating systems use to manage RAM. Less wasted memory means far more requests fit on the same GPU.

On top of that, vLLM adds continuous batching (new requests join the GPU's workload mid-flight instead of waiting for a batch to finish), prefix caching, and support for hardware from NVIDIA and AMD GPUs to Google TPUs. It also speaks an OpenAI-compatible API, so an app written for OpenAI's endpoints can point at your own server with a one-line change.

How it differs from llama.cpp: llama.cpp is tuned for one user on modest hardware, while vLLM is tuned for throughput — many users, big GPUs, production traffic. If you're self-hosting an open-weight model behind a real product, vLLM is usually the default choice; for multi-server clusters, the related llm-d project builds on top of it.

For the basics of what happens when a model answers a request, start with LLM inference.