Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity

TL;DR

<3-5 sentence high-level summary>MoE architectures often suffer structural isolation that leads to memory overwrite and knowledge fragmentation, especially in instruction-free settings. CDSP-MoE addresses this by grounding expert specialization in a shared physical subspace and using a Lagged Gradient Game to prune interfering connections, enabling emergent modularity without human task labels. The framework combines a Physical Subspace Backbone, aTopology-Aware Instantiation, adversarial masking to prevent shortcut routing, and a two-speed optimization regime to drive rapid topology evolution while preserving stable feature learning. Experiments in a heterogeneous multi-task vision setting demonstrate robust, content-driven routing under blind inference, with clear emergence of semantic clusters and improved modular disentanglement over baselines.</3-5 sentence summary>

Abstract

Mixture-of-Experts (MoE) architectures achieve parameter efficiency through conditional computation, yet contemporary designs suffer from two fundamental limitations: structural parameter isolation that causes catastrophic forgetting, and instruction-overfitting that degrades performance in instruction-free scenarios. We propose CDSP-MoE (Conflict-Driven Subspace Pruning MoE), a framework that addresses these issues through a paradigm shift from isolated expert containers to dynamic expert instantiation within a shared physical subspace. Grounded in the Universal Weight Subspace Hypothesis, CDSP-MoE maintains a super-complete parameter backbone where logical experts are carved out via learnable topology masks. Unlike prior work that uses gradient conflict for token reassignment or optimization surgery, we leverage it as a structural supervisory signal: a Lagged Gradient Game penalizes interfering connections in the shared manifold, enabling the topology to spontaneously prune conflicting pathways and evolve interpretable modular structures. Experimental results demonstrate that CDSP-MoE achieves robust content-driven routing without human-defined task labels, maintaining semantic specialization even under strict blind inference protocols where explicit instructions are absent. Code is available at: https://github.com/konodiodaaaaa1/Conflict-Driven-Subspace-Pruning-Mixture-of-Experts

Figures (9)

• Figure 1: The CDSP-MoE Framework.
• Figure 2: Evolution of the topology matrix $\sigma(\mathbf{A})$ across training epochs.
• Figure 3: Comparison of expert utilization heatmaps between Epoch 2 and Epoch 9.
• Figure 4: Routing distribution of CDSP-MoE under blind inference (Task ID = None).
• Figure 5: Routing distribution of the Standard MoE Baseline in the training state.