The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts

TL;DR

The paper investigates how MoE granularity—the number of active experts per token—affects expressivity, revealing an exponential separation as granularity increases while keeping active parameters fixed. It develops a formal theory for constant, linear, and ReLU activations, identifying the combinatorial quantity $\binom{m}{k}$ as the driver of expressivity, and constructs gating and expert configurations to prove inapproximability results when $\binom{m'}{k'}$ is comparatively small. Theoretical results are complemented by experiments showing that learning a teacher MoE requires matching granularity, supporting the practical relevance of the theory for frontier architectures. The work suggests that future MoE designs should embrace higher granularity and motivates the development of routing schemes capable of efficiently supporting it, with attention to hardware and routing overheads.

Abstract

Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total parameter count. This paper investigates the impact of the number of active experts, termed granularity, comparing architectures with many (e.g., 8 per layer in DeepSeek) to those with fewer (e.g., 1 per layer in Llama-4 models). We prove an exponential separation in network expressivity based on this design parameter, suggesting that models benefit from higher granularity. Experimental results corroborate our theoretical findings and illustrate this separation.

Figures (3)

• Figure 1: An intuitive picture to keep in mind when interpreting Theorem \ref{['thm:informal-main']}. Imagine expert models that have just enough parameters to answer questions on one of three different topics (Math, Biology, Business) in one of three different languages (English, French, Korean). In order to support all combinations of topics and languages, an MoE model with granularity 1 requires 9 experts -- one for each {topic, language} combination. On the other hand, an MoE model with granularity 2 can support all combinations with only 6 experts (3 for topics, and 3 for languages), since higher granularity allows for parameter reuse and thereby for more parameter-efficient models.
• Figure 2: Each data point is the test loss of a teacher MoE trained to learn a student MoE.
• Figure 3: We fix a 16-expert 8-active teacher model, and train student models with varying granularities and total number of parameters. Note that even with up to 16 times as many total parameters, student models do not fit the teacher unless their granularity is at least 8.

Theorems & Definitions (73)

• Remark 2.1: Variations on linear routing architecture
• Remark 2.2: Number of active and total parameters
• Theorem 3.1: Benefits of granularity; constant activation
• Lemma 3.2: Routing vectors
• proof : Proof sketch
• Lemma 3.3: Construction of constant experts for $\sigma(t) = 1$
• proof : Proof sketch
• proof : Proof of Theorem \ref{['thm:sep-constant']}
• Theorem 3.4: Benefits of granularity; linear activation
• Lemma 3.5: Approximating linear functions over large-volume sets; Lemma \ref{['lem:helper-10']}