How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts

TL;DR

MASS tackles the problem of allocating semantic capacity in Sparse Mixture-of-Experts by introducing a gradient-based semantic drift detector and semantic alignment checks to trigger adaptive expert expansion. It integrates a Top-$p$ routing strategy to balance per-token routing confidence and computational cost. The approach is validated on synthetic language-like data and real-world language and vision benchmarks, where MASS consistently outperforms strong MoE baselines and demonstrates improved semantic specialization and domain robustness. By reducing reliance on hyperparameter searches for the expert pool size and promoting diverse, complementary expert roles, MASS offers a practical path to efficient, scalable, semantically aware MoE systems.

Abstract

Finding the optimal configuration of Sparse Mixture-ofExperts (SMoE) that maximizes semantic differentiation among experts is essential for exploiting the full potential of MoE architectures. However, existing SMoE frameworks either heavily rely on hyperparameter tuning or overlook the importance of diversifying semantic roles across experts when adapting the expert pool size. We propose Mixture-of-Experts for Adaptive Semantic Specialization (MASS), a semanticaware MoE framework for adaptive expert expansion and dynamic routing. MASS introduces two key advancements: (i) a gradient-based semantic drift detector that prompts targeted expert expansion when the existing expert pool lacks capacity to capture the full semantic diversity of the data, and (ii) an integration of adaptive routing strategy that dynamically adjusts expert usage based on token-level routing confidence mass. We first demonstrate that MASS reliably converges to the point of optimal balance between cost-performance trade-off with notably improved sematic specialization in a highly controlled synthetic setup. Further empirical results on real-world datasets across language and vision domains show that MASS consistently outperforms a range of strong MoE baselines, demonstrating its domain robustness and enhanced expert specialization.

Figures (4)

• Figure 1: Comparison of Adaptive Expert Expansion Mechanisms. Left (DynMoE): Experts are added when tokens fail to activate any existing expert, indicating insufficient token coverage. This strategy relies on token-level activation statistics, without explicitly assessing semantic adequacy. Right (MASS): Expert expansion is triggered through a two-step process: (1) MASS statistically detects upward shifts in gradient norm over training steps to identify experts under high update pressure, and (2) it evaluates semantic drift by measuring misalignment between the gradient update $\Delta \mathbf{W}$ and the expert’s current representation $\mathbf{W}$. If the alignment $\|\cos(\Delta \mathbf{W}, \mathbf{W})\|$ falls below a threshold $\delta$, a new expert is instantiated to specialize on the diverging semantics.
• Figure 2: Synthetic Results for Identifying Optimal MoE Configurations with MASS. Left: Test loss vs. expert count for Naive and MASS. Gray dots represent individual runs of Naive with mixed top-$k$ across different $K$, and the solid blue curve connects the best performing run per $K$, forming the optimal frontier. Elbow represented by a gray vertical dashed line denotes the point of the most cost-effective $K$. Green dots show five MASS runs, and the purple star shows their average (Avg 12.4 experts), which aligns closely with the elbow. Center: Average test loss comparison between Naive with varying expert counts (gray bars) and MASS average (dashed green line), showing consistently lower loss for MASS (Avg 2.15). Right: Test loss vs. top-$k$ for fixed $K \in \{5, 10, 15, 20\}$. Colored curves represent per-$K$ optimal frontier for Naive. The 5 MASS samples from the left plot are shared here as well (green circles), with the average active experts marked by the purple star (Avg 3.9).
• Figure 3: Expert Specialization via Routing Divergence by Different Semantics. Contour plot comparing pairwise JSD of average routing distributions grouped by semantic labels ($x$=entity and $y$=property) defined in the synthetic data.
• Figure 4: Performance Comparison on GLUE benchmarks. The top five plots show top-1 accuracy for each GLUE task comparing MASS, DynMoE, and static MoE configurations. Gray dots connected by gray line represent the performance of fixed MoE baselines, with horizontal lines marking average performance of DynMoE (blue) and MASS (purple). Numbers indicate the average expert count $K$ and activated $k$ across three random seeds. The bottom-right figure provides summary comparison for top-1 accuracy, $K$, and $k$ in average across models.