Training versus inference
Training changes a model using examples. Inference uses the resulting model. Training may happen occasionally; inference happens every time a user asks for an answer, so inference often becomes the recurring operating bill of an AI product.
What changes inference cost
| Factor | What it changes |
|---|---|
| Model size | More capability can require more compute |
| Input and output tokens | Longer requests and answers consume more work |
| Latency target | Faster responses may require more expensive capacity |
| Traffic shape | Spikes need capacity or queueing |
| Hardware and hosting | Cloud, GPU, batching, and region affect unit economics |
Why product teams measure it
A prototype can feel free because only a few people use it. In production, you need cost per successful task, response time, failure rate, and capacity under peak load. Those numbers determine whether the feature is a business or an expensive demo.