Mixture of Expert

Mixture of Experts (MoE) uses several expert networks inside a model. In a sparse MoE layer, each input activates only a few experts.

Router and experts

Component	Job
Router, also called a gating network	Scores the experts and selects which ones process an input
Expert	Processes the input with its own learned weights
Weighted combination	Combines selected expert outputs using the router's weights

The router and experts learn during training. In Transformer MoE models, routing usually occurs separately for each token at each MoE layer.

How routing works

1. Read the token's current representation
2. Calculate a score for each expert
3. Select the highest-scoring k experts, called top-k routing
4. Send the representation to those experts
5. Combine their outputs using the routing weights

token representation -> router -> selected experts
                                      |
                                weighted outputs
                                      |
                                next model stage

Switch Transformer uses top-1 routing: one expert processes each token at a Switch layer. Switch Transformer paper

Where MoE fits in a Transformer

In models such as Mixtral, an MoE layer replaces the usual feed-forward block in a Transformer. Each expert is a feed-forward network, related to a multilayer perceptron. Attention remains a separate operation.

For example, Mixtral 8x7B has eight experts per layer and selects two for each token. Different tokens can use different experts. Mixtral paper

Total and active parameters

• Total parameters: all weights stored in the model, including every expert
• Active parameters: weights used to process a particular token, including the selected experts and shared parts

Adding experts increases model capacity while keeping the number selected per token small. All expert weights still require storage.

Practical limits

• Load balancing: spread tokens across experts to reduce uneven workloads
• Communication: moving tokens between experts on different devices adds cost
• Expert meaning: an expert is a learned network component; its name alone does not specify a subject such as mathematics or coding

MoE describes how computation is selected inside the model. Self-Attention describes how token representations use information from other positions.