Bigger models tend to perform better, but in a standard “dense” Transformer every parameter is used for every token, so compute grows with model size. Mixture-of-experts (MoE) models break that link: they contain many parameters but activate only a fraction of them for each token. The Switch Transformer paper by William Fedus, Barret Zoph and Noam Shazeer at Google, first released in 2021 and published in the Journal of Machine Learning Research in 2022, made this idea simple and scalable — and MoE has since become a common architecture for large language models.

The idea of mixture-of-experts

Mixture-of-experts dates back to the early 1990s: several specialized “expert” networks, plus a gating (or routing) network that decides which experts handle each input. In 2017, Shazeer and colleagues showed sparsely-gated MoE layers could massively scale language models, routing each input to a small number of experts among thousands.

What Switch Transformers changed

In a Transformer, each layer contains an attention block and a feed-forward block. The Switch Transformer replaces some feed-forward blocks with many expert feed-forward networks and a router.

The key simplification: route each token to just one expert (top-1 routing), rather than two or more as in earlier work. This:

  • reduces computation and communication between devices;
  • simplifies implementation;
  • still preserves model quality, as the paper showed.

Because each token uses only one expert, adding more experts increases parameters without increasing the computation per token.

Making it work

The paper introduced practical techniques to stabilize training:

  • an auxiliary load-balancing loss encouraging the router to spread tokens evenly across experts, so no expert is overloaded or idle;
  • an expert capacity limit on how many tokens each expert processes per batch, with overflow tokens skipping the layer;
  • selective precision, using higher precision in the router to avoid instability while keeping the rest in lower precision;
  • careful initialization and regularization, including dropout in experts during fine-tuning.

Results

The authors reported that Switch Transformers reached the same quality as dense T5 models with up to seven times faster pre-training for the same computational budget per token. They scaled to a model with about 1.6 trillion parameters, among the largest models at the time, and showed improvements carried over to fine-tuning on downstream tasks and to multilingual settings. They also showed large sparse models could be distilled into smaller dense ones while retaining part of the quality gain.

Trade-offs

  • Memory: all experts must be stored, so total memory is large even if compute per token is small.
  • Communication: experts are spread across devices, requiring tokens to be sent between them.
  • Training instability and the need for load balancing.
  • Fine-tuning challenges: sparse models can overfit more easily on small datasets.

Why it matters today

MoE has become a standard way to scale efficiently. Open models such as Mistral’s Mixtral and DeepSeek’s V3 family use mixture-of-experts architectures, and several frontier labs are widely reported to use MoE in their largest models. Modern designs vary — more, smaller experts, shared experts, top-2 or top-k routing — but they build on the principles that Switch Transformers made practical.

When you read that a model has a large “total” parameter count but a much smaller number of “active” parameters per token, you’re looking at the legacy of this line of work.

An intuition

Think of a large hospital: every patient doesn’t see every specialist. A triage nurse (the router) sends each patient to the relevant specialist (an expert). The hospital can employ many specialists — a large total capacity — while each patient’s visit takes roughly the same time. The difficulty is making sure no specialist is overwhelmed while others sit idle, which is what load balancing addresses.

What to watch next

  • Fine-grained expert designs with many small experts.
  • Efficient serving of MoE models on limited hardware.
  • Routing methods that improve specialization and stability.
  • MoE in multimodal and on-device models.

Key takeaways

  • Mixture-of-experts activates only some parameters per token.
  • Switch Transformers simplified MoE by routing each token to a single expert.
  • Load-balancing losses and expert capacity limits stabilized training.
  • The authors reported up to seven times faster pre-training than comparable dense T5 models and scaled to 1.6 trillion parameters.
  • MoE is now common in large open and frontier language models.

Frequently asked questions

What are “active parameters”?

In an MoE model, the parameters actually used to process each token, as opposed to the total parameters stored.

Are MoE models cheaper to run?

They need less computation per token than dense models of similar total size, but still require memory to store all experts.

Do experts specialize in topics?

Experts tend to specialize in patterns such as types of tokens or syntax, not necessarily human-recognizable topics.