Mixture of experts (MoE) explained simply

Mixture-of-experts models have many parameters but use only a fraction for each token. How routing works, why it makes large models cheaper to run, and the trade-offs engineers face.

A mixture-of-experts (MoE) model replaces some transformer layers with several parallel sub-networks, called experts, plus a small router that sends each token to only a few of them. The model holds a very large number of parameters but uses only a fraction per token, making it cheaper to run than a dense model of the same size.

MoE is one of the main reasons recent large models became faster and cheaper without becoming less capable. The core idea is old and simple; making it work at scale took years of engineering.

What problem does MoE solve?

In a standard dense transformer, every parameter is used for every token. Making the model bigger makes every token more expensive. MoE breaks that link: you add capacity by adding experts, but each token only visits a few, so compute per token grows much more slowly than total parameters.

How does routing work?

The idea of sparsely gated experts in neural networks was developed in research on outrageously large neural networks and scaled in transformers by work such as Switch Transformers.

Because the router itself is trained along with the experts, the model learns which experts suit which kinds of tokens. If you are new to the transformer layers that MoE modifies, Program Zero’s explainer on how LLMs work gives the context.

What does a real example look like?

In Mixtral 8x7B, each layer has eight expert feed-forward blocks and a router selects two experts for every token. Its authors report that each token has access to 47 billion parameters but uses only 13 billion active parameters during inference. That is the essence of MoE: large capacity, smaller active compute.

What are the trade-offs?

How is an MoE model trained?

Training follows the usual next-token objective, plus extra terms that encourage the router to spread tokens across experts. Engineers also limit how many tokens each expert can take in a batch, so no expert is overloaded. When experts live on different GPUs, the training system must move tokens between devices efficiently, which is a significant engineering challenge in its own right.

Is MoE always better than a dense model?

No. For small models or tight memory budgets, a dense model is often simpler and just as good. MoE pays off at larger scales, where saving compute per token matters most and there is enough memory to hold every expert. Like most architecture choices, it is a trade-off judged by cost, latency and quality on real tasks.

Why does MoE matter for developers?

Model cards increasingly list both total and active parameters. Understanding MoE helps you read those numbers: an MoE model may need as much memory as a much larger dense model, while running at a speed closer to a smaller one. That affects hardware planning, cost estimates and whether a model can be self-hosted.

How do you read an MoE model card?

Look for three numbers: total parameters, active parameters per token, and the number of experts with how many are selected. Total parameters tell you the memory you need; active parameters tell you roughly how fast and cheap each token will be; the expert count tells you how the model is structured. Then check benchmark results and, as always, test on your own tasks before choosing it.

How does MoE relate to other efficiency techniques?

MoE reduces compute per token; quantisation reduces memory per parameter; caching and batching improve serving throughput. Production systems often combine all of them. See our guides to LLM quantisation and how LLM inference works.

Where can you learn modern architectures?

Program Zero’s advanced LLM engineering phase covers mixture of experts with routing and load balancing, alongside RoPE, grouped-query attention and long-context techniques, and reading and implementing components from recent open-model papers. It builds on the transformer fundamentals taught earlier.

Program Zero runs live online, so you can learn this from anywhere in India: metros such as Chennai, Hyderabad, Bengaluru and Pune, growing cities such as Coimbatore, Kochi, Indore and Nagpur, and smaller towns such as Ranchi, Patna, Bhubaneswar and Raipur. Every learner joins the same evening classes, at the same fee.

Frequently asked questions

Related reading