Quantization vs. Training Precision: What 'Smaller' Really Means
Quantization and training precision get treated as the same thing when a vendor says a model is "smaller" — they aren't. Training precision (BF16, increasingly FP8) is the number format baked in while a model learns; post-training quantization (INT8, INT4) compresses an already-trained model afterward for cheaper serving. The distinction matters more now that FP8 training and aggressive quantization have become standard levers for cutting inference cost. For PMs, knowing which one you're buying changes what quality tradeoff you should actually expect — and what you should test before signing off.
Do you actually know which of the three levers your vendor pulled to call their model "smaller" — fewer parameters, lower training precision, or post-training compression? Most PMs can't answer that, and it's the reason quality regressions get discovered in production instead of in evaluation. Each lever fails differently: distillation loses breadth, aggressive quantization loses precision unevenly across tasks, and neither shows up cleanly in a generic benchmark score the vendor hands you. This week: pull the model card or ask your vendor contact directly whether the "efficient" or "small" tier you're using (or considering) was quantized post-training, and if so, at what bit width and calibrated on what data. Then run your own eval set — not theirs — against both the full-precision and compressed versions on the specific task your product depends on most. To be fair, most vendors aren't hiding this to deceive you — they're optimizing for a marketing word that tests well, and the underlying cost savings are real and worth using. But real savings don't excuse skipping the check. If you're shipping a compressed model into a workflow where accuracy actually matters — legal, medical, financial, anything customer-facing with real stakes — verify before you commit, not after a customer notices the model got worse.
Training precision (BF16, increasingly FP8 on Hopper/Blackwell GPUs) is set once during pretraining; post-training quantization (INT8, INT4, via AWQ/GPTQ) compresses an already-trained model afterward for serving.
Ask a vendor how they made their new model "smaller" and you'll get one of two very different answers — and most PMs can't tell which one they just got. One answer means the model was built smaller from the start, trained end-to-end in a lower-precision number format. The other means a full-size model got compressed after training, purely to make it cheaper to serve.
Same word, two engineering decisions, two different risk profiles for your product.
Here's the verdict: if you're evaluating a "smaller" or "efficient" model claim and haven't asked which one you're getting, you're buying blind. It's not that quantized models are worse; it's that "smaller" is doing the work of three separate engineering decisions at once, and only one of them is the thing you should actually be pricing risk against.
Training precision is the format the model was built in
Training precision is the numeric format used while the model learns — during pretraining and fine-tuning. Most large models today are still trained in BF16 by default, a 16-bit float format that balances range and stability. A smaller but growing set are trained natively in FP8, an 8-bit float format that Hopper- and Blackwell-class GPUs support directly in hardware.
DeepSeek's V3 was one of the earlier large open-weight models trained natively in FP8 rather than the more common BF16, and it's been widely cited since as evidence the approach scales.
This choice happens once, baked into the weights themselves. It shapes what the model actually learned to represent, not just how it's packaged for serving later.
Post-training quantization is compression after the fact
Post-training quantization (PTQ) takes an already-trained model and compresses its weights down to a lower-precision format — typically INT8 or INT4, often via techniques like AWQ or GPTQ — purely to cut memory and speed up inference. No retraining happens. The model's learned behavior is approximated, then calibrated against a sample dataset to limit the damage.
Think of it like the difference between shooting on a lower-resolution sensor versus shooting in full resolution and compressing the file afterward. Training precision is the sensor. Quantization is the JPEG compression you apply once the photo already exists.
Both make the file smaller. Only one of them changes what the camera was capable of capturing in the first place.
The numbers back up why teams do this. FP8 inference on Hopper-class GPUs has been reported to run roughly a third faster in tokens per second than FP16, with quality loss described as near-negligible on standard benchmarks. INT8 quantization typically halves VRAM requirements outright.
Push to INT4 and a 70-billion-parameter model that needs on the order of 140GB in FP16 can shrink to somewhere in the 35–40GB range — the difference between needing a rack of GPUs and needing one.
Why vendors blur the line, and why that's not entirely their fault
Vendor marketing bundles three distinct levers — fewer parameters (distillation), lower training precision, and post-training quantization — under one comforting word: "efficient." A model card that says "our new small model" could mean any combination of the three, and the quality tradeoff you should expect is completely different depending on which one it is.
To be fair to them: quantization and FP8 training are genuinely why inference got cheap enough to build on. Most production LLM traffic today already runs through some quantized path, and the cost collapse it enabled is real, not marketing. The problem isn't the technique.
It's that "smaller" gets used as a single label for decisions with very different failure modes — and PTQ, especially at INT4, degrades unevenly across tasks in a way that a headline benchmark score won't show you.
A checklist before you sign off on a "quantized" or "efficient" offering
- Ask directly: was this model trained in lower precision, or is it a full-precision model quantized afterward for serving? Get the answer in writing, not a marketing deck.
- Request quality benchmarks on tasks resembling yours, not generic leaderboard scores — quantization degrades unevenly, and reasoning-heavy or long-context tasks tend to feel it first.
- Ask what the PTQ calibration set was. A model calibrated on general web text and quantized down to INT4 can lose accuracy fastest on your specific domain, not the vendor's test set.
- Find out if quantization-aware training (QAT) was used versus naive post-hoc PTQ. QAT bakes the compression into training and typically holds up better under aggressive compression.
- Check whether "smaller" also means fewer parameters (distillation). That's a third, separate axis stacked on top of precision, and it compounds the quality question.
- Run your own eval set against the quantized version before committing spend, not just the vendor's demo.
The harder question underneath all of this isn't which vendor to pick. It's whether you standardize your own inference stack around one precision format across every vendor you use, or let each one ship its own definition of "smaller" and eat the cost of reconciling quality claims that were never measured the same way twice.
Get this in your inbox. AI Rundown Daily delivers original briefings every morning — free. Subscribe →
Frequently Asked Questions
Because benchmark scores are usually generic and your workload isn't. A model quantized to INT4 can hold up fine on MMLU-style multiple choice while losing meaningful ground on long-context reasoning or your specific domain, since PTQ degrades unevenly across task types. Knowing whether you're looking at a quantized model versus one trained natively in low precision tells you where to look for that degradation before you find out from a customer complaint.
Reported ranges put INT8 at roughly halving VRAM requirements versus full precision, and INT4 cutting a large model's footprint by more than half again, which directly lowers per-token serving cost and often improves latency too. Whether it's worth the effort depends on whether your task tolerates the accuracy tradeoff — high-volume, lower-stakes workflows (classification, summarization, drafting) are usually safe bets; anything with narrow correctness margins needs your own eval pass first.
The realistic failure isn't a dramatic outage — it's a slow accuracy drift that shows up as more edge-case errors, worse handling of ambiguous inputs, or subtly wrong outputs in a domain the vendor's calibration set didn't cover well. That's harder to catch than an API error and easier to blame on prompt quality when the real cause is the compression. The fix is boring but effective: keep a held-out eval set from your own domain and rerun it any time a vendor changes precision or quantization under the hood, even if the model name doesn't change.
