Why Mixtral 8x7B's 47B Parameters Don't Mean What You Think They Mean

47B parameters.
Only about 13B of them are doing the heavy lifting for a token.
That is the part of Mixture of Experts that sounds wrong the first time you hear it.
Take Mixtral 8x7B.
It has 8 expert networks.
But a token doesn't get sent through all 8.
A router picks 2 experts for that token.
So:
47B parameters in the model
~13B active for a token
The rest sit out that particular computation.
And the router makes a different choice for the next token.
So the model might send one token to Expert 2 + Expert 7...
then the next token to Expert 1 + Expert 5.
The experts aren't manually assigned.
Nobody sits there and says:
Expert 1 = Python
Expert 2 = French
Expert 3 = Mathematics
The model learns how to specialize those networks during training.
That gives you something interesting:
A model can have a much larger total parameter capacity without making every token pay the compute cost of the entire model.
But there is a price.
Those unused parameters still have to live somewhere.
In a distributed setup, the router may need to send token representations to different GPUs.
Some experts can become overloaded.
Communication between devices becomes expensive.
Load balancing suddenly matters.
So MoE isn't:
"Build a huge model and magically make it cheap."
It's closer to:
"Build a huge model and learn which parts are worth using for each token."
That's why parameter count alone doesn't tell you how expensive a model is to run.
Two models can have very different total parameter counts...
and surprisingly similar compute per token.
That distinction becomes extremely important once you start looking seriously at modern LLM architectures.

46.7B total / 12.9B active parameters — confirmed by Mistral AI's official blog and the Mixtral paper. Rounding to "47B" and "~13B" is fine.

8 experts, router selects top-2 per token — confirmed. This is the exact mechanism described in Mistral's own release.

Router choice varies token-to-token, no manual expert-to-domain assignment — confirmed, and correctly nuanced: the paper itself notes expert routing actually correlates more with syntax than with topic/domain (e.g., not "Expert 1 = Python"), which the post gets right by explicitly debunking that intuition.

Not every parameter replicated 8x — worth a small precision note the post skips: only the FFN layers are duplicated per expert; attention layers are shared across experts, which is why total params land at ~47B rather than 8×7B=56B. The post doesn't get this wrong, it just doesn't explain it — a minor simplification, not an error.

"Unused parameters still have to live somewhere" / memory footprint stays at full model size — confirmed. Multiple sources note that even though only 12.9B params are active in compute, all 46.7B must stay resident in memory, which is exactly the point the post is making.

Distributed routing, GPU communication overhead, load balancing being a real engineering problem in MoE — accurate and well-established characterization of MoE deployment challenges, not exaggerated.

Core thesis ("total params ≠ compute cost") — this is the correct, well-supported takeaway and matches how Mistral itself frames it ("processes input... at the same speed and cost as a 12.9B model").