What is LLM quantization?

Quantization is a technique that shrinks an LLM by storing its numbers at lower precision, making the model smaller and faster to run with only a small loss in quality. An LLM is, at heart, billions of numbers (parameters). Normally each is stored in a fairly precise format that takes a lot of memory, and quantization rounds these numbers to a coarser format — say, from 16 bits down to 8 or even 4 bits each.

Why does lower precision make a model smaller and faster?

Fewer bits per number means the model takes up less memory and has less data to shuffle around, so it runs faster and fits on more modest hardware. The payoff is practical reach: a quantized model can run on cheaper hardware, sometimes even a laptop or phone, instead of demanding a rack of expensive GPUs. That's a major reason people can now run capable open models locally instead of only through a cloud service.

What's the trade-off?

The cost is a slight drop in accuracy, since rounding throws away a little detail — but modern quantization methods are good enough that the loss is often barely noticeable for everyday use. Think of it like compressing a high-resolution photo: the file gets much smaller, and for most purposes you can't tell the difference. The more aggressive the compression, the more quality you risk losing, so it's a balance between size and fidelity rather than a free lunch.

What are the common precision levels?

Quantization is usually described by how many bits each number uses. The common steps are:

• 16-bit — a typical full-precision starting point: best quality, largest size.
• 8-bit — roughly half the memory, with very little noticeable quality loss.
• 4-bit — a much smaller footprint that makes local use practical, at the cost of a small but usually acceptable quality hit.

Quantization is one of the main tricks, alongside distillation, that makes running LLMs affordable at scale.