What is LLM distillation?

Distillation is a technique for training a smaller, cheaper LLM to mimic a larger, more capable one — capturing much of the big model's skill in a fraction of the size. The large model is called the teacher, and the small one is the student. Instead of training the student only on raw text, you also train it on the teacher's outputs — its answers, and often the detailed probabilities behind them.

The student learns to imitate how the teacher responds, effectively absorbing its behavior.

How does distillation actually work?

The key move is what the student learns from. Ordinary training shows a model raw text and asks it to predict the next word. Distillation adds a richer signal — the teacher's own output for each example, including the full spread of probabilities it assigned across possible answers, not just the single word it landed on.

That spread carries a lot of information: it tells the student not only what the teacher said but how confident it was and what it nearly said instead. Learning to match that softer, more detailed target is what lets a small model soak up a surprising amount of a big model's behavior.

Why does distillation matter?

The payoff is a compact model that runs faster and cheaper while keeping much of the teacher's quality. That's a big deal wherever the cost and speed of the biggest models would be prohibitive:

• Lower cost — a smaller model is far cheaper to run at scale, across millions of requests.
• Faster responses — fewer computations per answer means lower latency.
• Runs on modest hardware — paired with quantization, distilled models can run on phones and laptops, not just data centers.

Many of the smaller, efficient models people use in production are distilled from larger flagships.

What are the limits of distillation?

Distillation isn't magic. The student rarely fully matches the teacher — some capability is always lost in the shrink — and it inherits the teacher's blind spots and biases along with its strengths. A distilled model can't outgrow what its teacher knew.

Even so, it's one of the main reasons genuinely useful AI can run cheaply and widely rather than only in giant data centers. Think of it as the AI equivalent of an expert training an apprentice to handle most of the everyday work.