What is Mixture of Experts?

Mixture of Experts (MoE) is an architecture where a model contains many 'expert' sub-networks, but only a few are activated for each token. A router network decides which experts handle which input.

A model might have 100B+ total parameters but only use 15B per token — getting near-big-model quality at a fraction of the inference cost. Mixtral, DeepSeek-V3, and GPT-4 (reportedly) use MoE.

Trade-offs: MoE models need more total memory (all experts must be loaded) and can be trickier to fine-tune, but they're the dominant design for efficient frontier models.