Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures

TL;DR

The paper tackles robustness in Sparse Mixture of Experts by introducing SymphonySMoE, which endows SMoE with an expert-to-expert social graph to coordinate routing. Framed as a probabilistic graphical model, gating is reinterpreted as posterior inference, and a learned adjacency between experts guides token routing, improving both accuracy and resilience to data contamination. The authors prove theoretical guarantees about co-selection of high-confidence expert pairings under perturbation and validate the approach with extensive experiments across language modeling (WikiText-103, C4), visual instruction tuning, and GLUE fine-tuning, scaling up to 7.4B parameters. The modular design preserves compatibility with existing SMoE architectures and incurs minimal overhead, offering a scalable path to robust, large-scale SMoE deployments.

Abstract

Sparse Mixture of Experts (SMoE) has emerged as a promising solution to achieving unparalleled scalability in deep learning by decoupling model parameter count from computational cost. By activating only a small subset of parameters per sample, SMoE enables significant growth in model capacity while maintaining efficiency. However, SMoE struggles to adapt to distributional shifts, leading to reduced robustness under data contamination. In this work, we introduce SymphonySMoE, a novel family of SMoE that introduces a social graph to model interactions among experts. This graph-based structure enhances the token routing process, addressing the robustness challenges that are inherent in conventional SMoE designs. SymphonySMoE is lightweight, modular, and integrates seamlessly with existing SMoE-based models such as the XMoE and the Generalist Language Model. We provide both theoretical analysis and empirical evidence demonstrating SymphonySMoE's advantages over baseline SMoE. Extensive experiments on language modeling and visual instruction tuning validate our method's effectiveness. We further highlight the scalability of SymphonySMoE to models with 4.2 and 7.4 billion parameters, showcasing its applicability in fine-tuning tasks for large-scale systems.

Figures (5)

• Figure 1: Graphical model underlying (A) MoE routing and (B) SymphonyMoE routing.
• Figure 2: An overview figure that illustrates the architecture of SymphonySMoE.
• Figure 3: Plots of 14x14 image reconstructions where each patch is colored by its assigned expert in the Swin-MoE (baseline) and SymphonySwin-MoE model. For each triple of images, the left-most image is the actual input to each model, the middle image is a visualization of the assigned expert for each patch in the baseline Swin-MoE while the right-most image is the corresponding plot for the SymphonySwin-MoE model.
• Figure 4: Evolution of social graphs across layers in Switch Transformers during Wikitext-103 pretraining.
• Figure 5: Expert selection frequencies for baseline SMoE and SymphonySMoE on WikiText-103, averaged across all layers. Experts are sorted by frequency; a more uniform distribution reflects better load balancing.

Theorems & Definitions (11)

• Remark 1: Deriving Other Routers
• Remark 2: (S)MoE as a Social Recommender System
• Remark 3: Connection of Algorithm \ref{['alg:symphony-routing']} and the Hebbian Learning Rule
• Definition 1: Symphony Sparse Mixture of Experts
• Remark 4: Symphony Router
• Remark 5
• Theorem 1
• Remark 6: Consistency
• Remark 7: Promotion of High-Confidence Pairings
• Proposition 1