Adaptive Graph Mixture of Residual Experts: Unsupervised Learning on Diverse Graphs with Heterogeneous Specialization

TL;DR

ADaMoRE tackles the challenge of learning adaptable graph representations across diverse graphs in an unsupervised setting by introducing a backbone-residual mixture of heterogeneous GNN experts. A structurally-aware gating module generates two complementary views of the graph and a sparse MoE backbone selects stable foundational processors, while a dense residual expert ensemble captures diverse computational patterns; an information-theoretic diversity regularizer enforces specialization, and an adaptive fusion gate combines views node-specifically. The framework is trained with a unified objective combining masked feature reconstruction, cross-filter self-supervision, and load-balancing regularization, achieving state-of-the-art results on unsupervised node classification and few-shot learning across 16 benchmarks and exhibiting faster convergence and better stability. The approach reduces sample complexity relative to naive heterogeneous MoEs by decomposing learning into a structured backbone and a smaller residual space, enabling data-efficient learning without supervision. This work advances robust, scalable unsupervised graph representation learning with practical impact on large-scale, diverse graph data where labels are scarce.

Abstract

Graph Neural Networks (GNNs) face a fundamental adaptability challenge: their fixed message-passing architectures struggle with the immense diversity of real-world graphs, where optimal computational strategies vary by local structure and task. While Mixture-of-Experts (MoE) offers a promising pathway to adaptability, existing graph MoE methods remain constrained by their reliance on supervised signals and instability when training heterogeneous experts. We introduce ADaMoRE (Adaptive Mixture of Residual Experts), a principled framework that enables robust, fully unsupervised training of heterogeneous MoE on graphs. ADaMoRE employs a backbone-residual expert architecture where foundational encoders provide stability while specialized residual experts capture diverse computational patterns. A structurally-aware gating network performs fine-grained node routing. The entire architecture is trained end-to-end using a unified unsupervised objective, which integrates a primary reconstruction task with an information-theoretic diversity regularizer to explicitly enforce functional specialization among the experts. Theoretical analysis confirms our design improves data efficiency and training stability. Extensive evaluation across 16 benchmarks validates ADaMoRE's state-of-the-art performance in unsupervised node classification and few-shot learning, alongside superior generalization, training efficiency, and faster convergence on diverse graphs and tasks.

Figures (8)

• Figure 1: The "one-size-fits-all" paradigm of GNNs is suboptimal. The ideal GNN configuration is highly context-dependent, varying with (a) local structural context, (b) node topological roles, and (c) the specific downstream task.
• Figure 2: Overview of the proposed ADaMoRE framework. A Structurally-Aware Gating module first generates two complementary structural views. These views are then processed by parallel Backbone-Residual MoE Channels. A final Adaptive Fusion Gating module combines the enhanced outputs. The framework is trained via a step-wise optimization, alternating between a primary MAE objective for the main model and a dedicated cross-filter reconstruction loss for the view generator.
• Figure 3: Ablation on expert number and diversity.
• Figure 4: Training stability vs. naive stacking.
• Figure 5: Performance vs. efficiency.

Theorems & Definitions (4)

• Lemma 1: Catastrophic Sample Complexity of Naive Heterogeneous MoE
• Remark 1
• Lemma 2: Sample Complexity Reduction via Backbone-Residual Architecture
• proposition 1: Efficacy of Guided Complementary Filtering as a Backbone