Mixture of Experts (MoE)

What Is Mixture of Experts (MoE)?

A dense model runs every parameter on every token. An MoE model has many expert sub-networks and a router that activates only a few per token, so a model with a large total parameter count runs at the cost of a much smaller one. Most frontier 2026 models use MoE. The trade-off is added routing complexity and memory to hold all experts even though only some fire.