Training Diverse Graph Experts for Ensembles: A Systematic Empirical Study

TL;DR

This work systematically studies how to train diverse graph experts to improve Graph Neural Network MoE ensembles for node classification. By evaluating 20 diversification strategies across 14 datasets within a train-then-merge MoE framework, it reveals that expert diversity enhances ensemble performance, with training-data partitioning offering the strongest gains and introducing mechanistic insights via Direction Informativeness (DI) and intra-class graph metrics. The findings show directional modeling yields limited improvements and that carefully chosen domain partitions—especially those based on intra-class properties and neighbor agreement—substantially boost performance, achieving up to several percentage points of improvement over the best single model. The study provides practical guidance for building effective graph MoEs and contributes conceptual tools for understanding when and why diversification helps in graph learning, with open-source code available for replication.

Abstract

Graph Neural Networks (GNNs) have become essential tools for learning on relational data, yet the performance of a single GNN is often limited by the heterogeneity present in real-world graphs. Recent advances in Mixture-of-Experts (MoE) frameworks demonstrate that assembling multiple, explicitly diverse GNNs with distinct generalization patterns can significantly improve performance. In this work, we present the first systematic empirical study of expert-level diversification techniques for GNN ensembles. Evaluating 20 diversification strategies -- including random re-initialization, hyperparameter tuning, architectural variation, directionality modeling, and training data partitioning -- across 14 node classification benchmarks, we construct and analyze over 200 ensemble variants. Our comprehensive evaluation examines each technique in terms of expert diversity, complementarity, and ensemble performance. We also uncovers mechanistic insights into training maximally diverse experts. These findings provide actionable guidance for expert training and the design of effective MoE frameworks on graph data. Our code is available at https://github.com/Hydrapse/bench-gnn-diversification.

Figures (13)

• Figure 1: Left: The relationship between accuracy gain (relative to the mean expert accuracy) and error inconsistency (EI) in the Amazon-Ratings dataset. Each datapoint corresponds to a pair of GNNs diversified using a specific strategy. We report results from three ensemble strategies: Upper Bound, Moscat Gating Ensemble (MoE), and Global-Ensemble (Ensemble). Right: Boxplots of EI across the four expert diversification strategy categories.
• Figure 2: Pairwise Error Inconsistency (EI) and Complementary Gain (CG) trend with a single modified hyperparameter (averaged over 5 random seeds). The datapoints are computed between a pair of experts: a tuned best-performing expert as baseline and a single-hyperparameter variant on the baseline.
• Figure 3: Pairwise error inconsistency across architectural design dimensions. Each heatmap displays the average error inconsistency (computed over 10 runs) between expert pairs that differ on a single architecture design. Each dataset in the same row shares a color scale.
• Figure 4: The label distribution and model performance across domains. Top: Label distribution (i.e., varied label counts per bin) for full graph metrics. Middle: Label distribution for intra-class graph metrics. Bottom: Performance of GCN model (use full training set) across different test domains. Domains (node subgroups) partitioned by intra-class metrics (rather than full-graph metrics) exhibit more significant label distribution shifts and greater disparities in expert generalization.
• Figure 5: Impact of domain assignment on average expert accuracy gain, comparing training from scratch vs. fine-tuning on domain-specific data. Gains are computed over a GNN trained on the full graph, averaged across datasets. Intra-class metrics yield the most effective assignments.