Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architectures

TL;DR

The paper tackles catastrophic interference in mixture-of-experts models by introducing Spectrally-Regularized MoE (SR-MoE), which enforces modular routing through spectral norm and stable rank penalties on the gating weights. By constraining the routing manifold to be Lipschitz-continuous and high dimensional, SR-MoE preserves gradient diversity and prevents expert collapse, enabling surgical, one-shot updates that affect only the relevant expert path. Across small to deep MoE architectures and varying data complexities, SR-MoE demonstrates superior structural integrity, near-zero one-shot interference in deep configurations, and meaningful positive transfer in some cases, outperforming baseline gating that collapses to a single expert. This framework provides a general, scalable approach for building high-capacity, modular networks capable of lifelong learning without catastrophic forgetting. The results highlight the practical potential for modular continual learning in vision tasks and beyond, by maintaining stable routing boundaries even as experts adapt to new tasks.

Abstract

Mixture of Experts (MoE) architectures enable efficient scaling of neural networks but suffer from expert collapse, where routing converges to a few dominant experts. This reduces model capacity and causes catastrophic interference during adaptation. We propose the Spectrally-Regularized Mixture of Experts (SR-MoE), which imposes geometric constraints on the routing manifold to enforce structural modularity. Our method uses dual regularization: spectral norm constraints bound routing function Lipschitz continuity, while stable rank penalties preserve high-dimensional feature diversity in expert selection. We evaluate SR-MoE across architectural scales and dataset complexities using modular one-shot adaptation tasks. Results show that traditional linear gating fails with increasing depth (accuracy drops up to 4.72% due to expert entanglement), while SR-MoE maintains structural integrity (mean interference -0.32%). Our spectral constraints facilitate positive knowledge transfer, enabling localized expert updates without global performance decay. SR-MoE provides a general solution for building high-capacity, modular networks capable of stable lifelong learning.

Figures (6)

• Figure 1: Deep SR-MoE Architecture. The model processes inputs through $N$ successive layers. In each layer, a bank of $K$ experts is available. The Spectral Regularization is strictly applied to the routing weights $\bm{W}_g$ in every layer to ensure manifold diversity. Surgical updates are performed by backpropagating only through the active expert chain (blue path).
• Figure 2: Expert distribution on the small dataset (N=525). The baseline shows complete path collapse (all data routed through a single expert). Clustering improves load balancing, and spectral clustering begins to separate semantic concepts into distinct expert pathways as detailed in Section \ref{['sec:one_shot_interference']}.
• Figure 3: Path utilization on the large dataset (N$\approx$ 1600). With sufficient data, the spectral model successfully utilizes the full architectural capacity ($2\times2$ experts), mapping each semantic class to a distinct expert circuit. Clustering shows improved load balancing over baseline, but lacks the structured specialization of spectral routing as detailed in Section \ref{['sec:one_shot_interference']}.
• Figure 4: Expert utilization patterns across routing methods (4×4 experts). (a) Baseline shows poor load balancing. (b) Clustering improves distribution but lacks structured specialization. (c) Spectral clustering provides both balanced load and semantic specialization, mitigating interference as analyzed in Section \ref{['sec:one_shot_interference']}.
• Figure 5: One-Shot Experimental Workflow. The model undergoes a surgical update using a single novel sample anchored by a training batch. The resulting Accuracy Delta ($\Delta$) quantifies the degree of catastrophic interference.