Mixture of Experts (MoE)
A Mixture-of-Experts model keeps many specialist sub-networks but activates only a few per token — big-model knowledge at small-model speed, with big-model memory still required.
Instead of one dense network that processes everything, an MoE model holds dozens or hundreds of smaller "experts" and a router that picks a handful per token. Labels like 35B-A3B say it directly: 35 billion parameters in total, about 3 billion active for any given token.
The split explains the appeal and the catch. Speed and compute cost follow the ACTIVE parameters — the model generates like a small one. Memory follows the TOTAL — all experts must be loaded, because the router may call any of them at any moment. An MoE is fast to run and heavy to hold.
For local use that means the download and the fit calculation care about the big number, while the feel of the model in use tracks the small one. Neither number alone describes the model; together they do.
Where you see it on the radar
MoE models on the board carry both numbers in their label — total and active — because your memory pays for one and your patience for the other.
Spot the MoE models on the board →Related terms