On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating

TL;DR

The paper provides a theoretical and empirical examination of two hallmark features of DeepSeekMoE: the shared-expert strategy and the normalized sigmoid gating. It establishes near-parametric convergence for exactly-specified shared-expert parameters under strong identifiability and demonstrates faster routed-expert estimation when using normalized sigmoid gating, contrasted with linear-expert scenarios where identifiability hinders rapid convergence. Through synthetic and real-data experiments in language and vision-language tasks, it confirms improved sample efficiency, faster convergence, and more stable router dynamics, including quicker router saturation and higher fairness in expert utilization. Together, these results offer principled guidance for designing sparse MoE architectures and gating mechanisms, with practical implications for scaling large models efficiently. Mathematical notation is used to characterize convergence rates, Voronoi losses, and identifiability conditions, highlighting the impact of shared-expert and gating structures on estimation accuracy and router dynamics.

Abstract

Mixture of experts (MoE) methods are a key component in most large language model architectures, including the recent series of DeepSeek models. Compared to other MoE implementations, DeepSeekMoE stands out because of two unique features: the deployment of a shared expert strategy and of the normalized sigmoid gating mechanism. Despite the prominent role of DeepSeekMoE in the success of the DeepSeek series of models, there have been only a few attempts to justify theoretically the value of the shared expert strategy, while its normalized sigmoid gating has remained unexplored. To bridge this gap, we undertake a comprehensive theoretical study of these two features of DeepSeekMoE from a statistical perspective. We perform a convergence analysis of the expert estimation task to highlight the gains in sample efficiency for both the shared expert strategy and the normalized sigmoid gating, offering useful insights into the design of expert and gating structures. To verify empirically our theoretical findings, we carry out several experiments on both synthetic data and real-world datasets for (vision) language modeling tasks. Finally, we conduct an extensive empirical analysis of the router behaviors, ranging from router saturation, router change rate, to expert utilization.

Figures (16)

• Figure 1: Average performance (%) over training steps in language modeling tasks. Left: Model with 158M parameters; Right: Model with 679M parameters.
• Figure 2: Average performance (%) over training steps on vision-language pretraining tasks. Left: Vanilla SMoE vs. DeepSeek-V3; Center: Vanilla SMoE vs. DeepSeek-V2; Right: DeepSeek-V2 vs. DeepSeek-V3.
• Figure 3: Evolution of router saturation (averaged across all layers) during training for language-modeling tasks with 158 M (left) and 679 M (right) parameter models. We compute saturation by comparing the routing to the top-8 experts with SMoE and SMoE Sigmoid Gating, and the top-6 experts with DeepSeek variants.
• Figure 4: Router Change Rate (averaged across all layers) during training for language-modeling tasks with 158 M (left) and 679 M (right) parameter models. We compute router change rate by comparing the routing to the top-8 experts with SMoE and SMoE Sigmoid Gating, and the top-6 experts with DeepSeek variants.
• Figure 5: Empirical illustration of the input - output relationship $(X, Y)$ under synthetic conditions for each theoretical result. Each subplot corresponds to a different theoretical setting: (a) Theorem 1, (b) Theorem 2, (c) Theorem 3, and (d) Theorem 4.

Theorems & Definitions (22)

• Proposition 1
• Definition 1: Strong Identifiability
• Theorem 1
• Theorem 2
• Proposition 2
• Theorem 3
• Corollary 1
• Definition 2: Weak Identifiability
• Theorem 4
• Proposition 3: Proposition 2.1, ho_convergence_2016