Why a 671-Billion-Parameter Model Doesn't Compute Like One: Understanding Mixture of Experts

A 671-billion-parameter AI model does not necessarily use 671 billion parameters to generate your next token. That sounds ridiculous until you understand Mixture of Experts. Imagine a normal dense Transformer. Every token enters a layer and goes through the same set of parameters. Whether the token is Python, mitochondria, or bonjour, the same feed-forward network does the work. Now make one change. Instead of one feed-forward network, put many of them inside the layer. Then add a tiny router whose job is to look at the current token representation and decide which few should handle it. Those networks are the "experts." The name makes them sound like someone manually created a coding expert, a French expert, a mathematics expert, etc. That isn't generally how it works. They are learned subnetworks, and whatever specialization emerges is learned during training. This gives you a strange machine. The model can contain an enormous number of parameters, while only activating a fraction of them for each token. DeepSeek-V3 is the cleanest example => 671B total parameters, about 37B activated for each token. So when it generates one token, hundreds of billions of parameters can effectively sit that computation out. That is the trick behind sparse MoE => increase the model's parameter capacity without paying the compute cost of running every parameter for every token. But there is a catch that gets missed constantly. Inactive does not mean nonexistent. Those weights still have to be stored somewhere. On large deployments they are spread across accelerators, and now the router has to send tokens to the right experts. Some experts can receive more traffic than others. Moving those token representations between devices creates communication overhead. Keeping expert utilization balanced becomes an engineering problem of its own. So a 671B MoE is not secretly a 37B model. It is a 671B model that has learned to choose which parts of itself are worth waking up for each token. And once you understand that, some of the strangest parameter counts in modern AI suddenly make a lot more sense.

A 671-billion-parameter model doesn't use all 671 billion parameters to generate your next token — and understanding why requires understanding Mixture of Experts.

In a standard dense Transformer, every token passes through the same feed-forward network in each layer, regardless of what the token is. Mixture of Experts changes this: instead of one feed-forward network per layer, there are many, and a small router examines each token's representation and decides which few of them should process it. These are the "experts" — not hand-assigned specialists in coding or French or math, but learned subnetworks whose specialization emerges naturally during training, often in less human-interpretable ways than the name suggests.

This produces models with enormous total parameter counts that activate only a fraction of those parameters per token. DeepSeek-V3 is the clearest real-world example: 671 billion total parameters, with only about 37 billion activated for any given token — meaning hundreds of billions of parameters sit out each individual computation.

That's the core trick of sparse MoE: it lets you scale a model's total capacity without paying the full compute cost of running every parameter on every token.

But there's a catch that's easy to overlook: inactive doesn't mean nonexistent. Those unused weights still have to be stored somewhere, typically spread across multiple accelerators in large deployments. The router then has to send each token to the right experts, and because some experts receive more traffic than others, moving token representations between devices creates real communication overhead — keeping expert load balanced becomes its own significant engineering challenge (DeepSeek-V3, for instance, uses an auxiliary-loss-free load-balancing strategy specifically to manage this).

So a 671B MoE model isn't secretly a 37B model. It's a 671B model that has learned which parts of itself are worth activating for each token — and once that clicks, some of the more unusual parameter counts in modern AI make a lot more sense.