AI Vocabulary · 01 The Model
What is Mixture of Experts?
A model built from many specialised sub-networks, of which only a few are switched on for each input.
A classic model uses all of its parameters for every token. A mixture of experts, MoE for short, splits part of the model into many sub-networks called experts. A small router decides, for each input, which few experts to switch on. The model can be very large in total while doing relatively little work per token.
The "experts" are not specialists in a human sense, like one for law and one for medicine. The model learns for itself how to split the work, and the split is usually not easy to interpret.
Why it matters for you
Mixture of experts makes inference cheaper per token, but the whole model still has to fit in memory. That is why some model makers state two numbers: total parameters and active parameters per token.
Own it
If you compare open-weight models for self-hosting, look at both numbers. Total parameters decide how much memory you need, active parameters roughly how fast and cheap each answer is.
Where it comes from
The idea of combining several expert networks is older, but the version used in today's large models goes back to Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer by Noam Shazeer and colleagues at Google (January 2017). Their layer had up to thousands of sub-networks and up to 137 billion parameters, and by the authors' account increased capacity more than a thousandfold with only minor losses in efficiency.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.