What is llama.cpp?

llama.cpp is a free, open-source engine written in C and C++ that runs large language models directly on your own hardware — a laptop, a mini PC, even a Raspberry Pi — with no cloud account required.

It started in 2023 when developer Georgi Gerganov got Meta's LLaMA model running on a MacBook, something most people assumed needed a rack of GPUs. The trick was aggressive quantization — shrinking a model's weights from 16-bit numbers down to 4 bits or fewer, trading a little quality for a huge drop in memory use.

A few things define it today:

• GGUF files. llama.cpp uses a single-file format called GGUF that bundles a model's weights, tokenizer, and metadata together. It has become the standard way local models are shared on Hugging Face.
• It runs on nearly anything. It works CPU-only and accelerates on Apple Metal, NVIDIA CUDA, AMD ROCm, Vulkan, and more.
• It's the engine under the hood. Friendlier tools like Ollama and LM Studio are built on top of llama.cpp — if you've ever run a local model with either, you've already used it.

The honest trade-off: llama.cpp itself is a command-line tool with a lot of knobs. Most people should start with the friendlier wrappers and drop down to llama.cpp when they want more control or better performance tuning.

Our guide to running an LLM locally walks through the whole toolchain.