Mixture of Experts Explained: The Math That Makes AI Cheaper
Mixture-of-Experts (MoE) is the architecture behind why frontier models keep getting bigger on paper while getting cheaper to run in practice. This piece breaks down total parameters versus active parameters per token, using DeepSeek-V3, Llama 4, Qwen3, and Kimi K2 as verified examples. The trend line matters more than any single spec: active-parameter share has been shrinking with every major release since 2023, decoupling capability from compute cost. For PMs, a vendor's headline parameter count is now closer to a marketing figure than a cost or performance signal, and knowing which number to ask for is a negotiating advantage.
When was the last time you actually asked a model vendor for their active-parameter count instead of nodding along at the total? Most procurement conversations still anchor on the total-parameter headline because it's the number in the press release, and that's exactly the number least correlated with what you'll pay per million tokens. This week, pull up your current LLM vendor contract or API pricing page and ask directly: is this model MoE, and if so, what's active per token versus total? If they won't say, that's itself useful information — it means you're pricing a black box. To be fair, some vendors have legitimate competitive reasons to withhold architecture details, and a refusal isn't automatically a red flag. But you can still get the information that matters practically: ask for their per-million-token pricing trend versus their last-generation model, and their published or benchmarked latency at your typical prompt length. That tells you the real cost curve regardless of what's happening inside the router. Do this before your next renewal conversation, not during it — vendors negotiate differently when you're asking informed questions from the start rather than catching up mid-call.
MoE splits a model into dozens or hundreds of expert sub-networks and routes each token to only a handful, so total parameters and active parameters per token are now two distinct, non-interchangeable numbers.
DeepSeek-V3 ships with 671 billion parameters. When it answers your prompt, only 37 billion of them ever switch on.
That's not a rounding error — it's the entire architecture. The gap between the spec-sheet number and the number that determines your inference bill has a name: Mixture-of-Experts, or MoE. And if you're still sizing up vendors by parameter count alone, you're reading the wrong line item.
Here's the thesis: MoE isn't a research curiosity anymore. As of mid-2026 it's the default architecture behind nearly every frontier model release, and it has quietly broken the old assumption that capability and inference cost scale together. Total parameters buy knowledge.
Active parameters buy the compute bill. Those two numbers have decoupled, and the gap keeps widening.
The Two Numbers That Actually Matter
A dense model — the GPT-3 era default — uses every one of its parameters on every token. Double the parameters, double the compute, every single request. MoE breaks that link by splitting the model's feed-forward layers into dozens or hundreds of "expert" sub-networks, then routing each token to only a handful of them.
The result is two very different numbers for the same model, and the gap between them has been widening with each generation:
| Model | Total params | Active params/token | Active share |
|---|---|---|---|
| Mixtral 8x7B (Mistral, 2023) | 47B | 13B | ~28% |
| GPT-4 (unconfirmed leak, SemiAnalysis) | ~1.8T | ~220B | ~12% |
| Qwen3-235B-A22B (Alibaba) | 235B | 22B | ~9% |
| DeepSeek-V3 (Dec 2024) | 671B | 37B | ~5.5% |
| Llama 4 Maverick (Meta, 2025) | 400B | 17B | ~4.3% |
| Kimi K2 (Moonshot AI) | 1T | 32B | ~3.2% |
Note the GPT-4 row is a caveat, not a confirmed spec — OpenAI has never disclosed it, and the widely-cited 1.8T/220B figures trace back to a 2023 SemiAnalysis report, not an official filing. But the trend across the confirmed open releases is unmistakable: newer models push active share *down*, not up. Labs are getting more aggressive about hoarding capacity while rationing compute.
The Routing Trick: Why Sparse Doesn't Mean Weak
Here's the part that trips people up: if only 3-5% of the model fires per token, why doesn't the model get dumber? Because a small gating network scores every expert for every token and routes it to the top few — usually top-2 or top-8 — that are actually relevant to that specific piece of text. A token about Python syntax and a token about protein folding get sent to different specialists.
Think of it like a large hospital. The hospital employs hundreds of specialists — cardiologists, oncologists, radiologists, the works — and that full roster is what makes it capable of treating almost anything that walks through the door. But when you show up with a broken wrist, you don't see all of them.
You see the two or three relevant to your case. The hospital's total headcount is its capacity; your bill only reflects the specialists who actually touched your case.
To be fair to the skeptics here: dense models aren't dead weight. They're more predictable under load, they don't suffer from routing imbalance (where some experts get overtrained and others starve for tokens), and at matched active-parameter budgets a well-tuned dense model can still edge out an MoE model on certain benchmarks. MoE is a cost-and-capacity trade, not a strictly-better-in-every-dimension upgrade.
Cheaper to Run Isn't the Same as Cheaper to Host
This is the nuance that trips up procurement conversations. Inference *compute* — the FLOPs, and therefore a big chunk of the per-query cost — tracks active parameters closely. But serving the model still requires holding all 671 billion, or 1 trillion, parameters in memory, because you don't know in advance which experts the next token will need.
It's not that MoE models are cheaper, full stop — it's that they shift the cost from compute to memory. A vendor advertising a "37B-active" model is telling you about their GPU compute bill, not their VRAM footprint, which is closer to the 671B number. If you're negotiating self-hosted deployment or a dedicated-capacity contract, the memory number is the one that determines your hardware floor, and it's usually the number vendors mention less loudly.
What to Actually Ask Your Vendor
Next time a vendor pitches you on "our new 1-trillion-parameter model," push past the headline number:
- What's the active-parameter count per token, and how many experts are selected (top-2? top-8?)
- Is the benchmark you're citing run at active-parameter parity against competitors, or total?
- What's the memory footprint for self-hosting or dedicated capacity, independent of the compute cost?
- Has the routing/load-balancing been validated at your expected traffic mix, not just benchmark traffic?
MoE's arrival as the default architecture isn't really in question anymore — it's the substrate under Mixtral, DeepSeek-V3, Qwen3, Llama 4, Kimi K2, and reportedly GPT-5-class models too, per industry analysis, though OpenAI won't confirm architectural specifics. The harder question for anyone building a roadmap on top of these models: once "parameter count" stops mapping cleanly to cost, capability, or hosting requirements, does it survive as a meaningful spec at all — or does it go the way of CPU gigahertz, a number everyone still quotes and almost no one can actually use to predict performance?
Get this in your inbox. AI Rundown Daily delivers original briefings every morning — free. Subscribe →
Frequently Asked Questions
Because it's the bigger, more impressive number, and total parameters do reflect real capacity — a model with more total parameters can generally store more knowledge and handle a wider range of specialized tasks. The problem is that this number tells you almost nothing about what you'll actually pay per query. As a PM, treat the total-parameter figure as a capability signal and the active-parameter figure as the cost signal, and don't let a vendor's marketing collapse the two into one number.
It genuinely reduces inference compute — a 37B-active MoE model costs roughly what a 37B dense model costs to run per token, while offering the knowledge capacity of a much larger model. The catch is memory: you still need enough VRAM or host memory to hold all 671 billion parameters (or however many the model has), because the router decides which experts fire on the fly, per token. So it saves on the compute line of your bill, not necessarily the hosting-hardware line, and self-hosted deployments need to budget for both separately.
The main documented failure mode is load imbalance — if the router consistently favors a subset of experts, those get overtrained while others go stale, which can show up as inconsistent quality across different prompt types. There's also less mature tooling and quantization support for MoE models compared to dense ones, which can complicate fine-tuning or on-prem deployment. And because architecture details are often undisclosed for closed models, you're frequently trusting a vendor's benchmark claims without being able to verify the underlying active-parameter math yourself.
