What is LLM inference?

Inference is what happens when an LLM actually runs to answer you — the moment it takes your prompt and generates a response. It's the counterpart to training. Training is the slow, expensive, one-time process of teaching the model by adjusting its billions of parameters.

Inference is using that finished model, over and over, to produce output. Every time you send a message to a chatbot, you're triggering an inference run.

How does inference actually work?

During inference the model doesn't learn or change; its parameters are frozen. It takes your input, processes it through its network, and predicts the answer one token — roughly one word-piece — at a time, feeding each new token back in to help pick the next. That's why the same model can give you fresh answers without ever remembering past chats unless the app deliberately feeds that history back into the prompt.

Every response is generated from scratch in the moment, which is also why a longer answer takes longer to produce.

Why does inference dominate AI costs?

Inference is where the real-world cost and speed of AI live. It needs powerful hardware, usually GPUs, and unlike training it never stops — it runs for every user, on every message, all day. Training is a big one-time bill; inference is a bill that arrives forever.

So when people talk about the price of running an AI product at scale, they're mostly talking about inference: not building the model, but serving it to millions of people, one prediction at a time.

How do companies make inference cheaper?

Because inference runs constantly, shaving a little off each request adds up fast. Common techniques include:

• Quantization — storing the model's numbers at lower precision so it uses less memory and runs faster.
• Caching — reusing computation for repeated or shared parts of a prompt instead of redoing it.
• Distillation — training a smaller, cheaper model to mimic a big one, trading a little quality for a lot of speed.

Together these let providers cut the cost per answer without retraining the model from scratch.