E2GNN: Efficient Graph Neural Network Ensembles for Semi-Supervised Classification

TL;DR

E2GNN tackles the challenge of ensemble learning for graph neural networks in semi-supervised node classification by distilling multiple GNN teachers into a single, fast MLP student. A node-level reinforcement-learning policy selects the most reliable teacher (or a null action) for each node, enabling selective distillation of soft labels and rejection of misleading predictions. The framework combines a labeled-node cross-entropy term with a KD term guided by teacher-student KL divergence, optimized via policy gradients with a validation-based reward. Empirical results across eight datasets and both transductive and inductive settings show that E2GNN consistently outperforms individual GNNs and existing ensemble baselines, while offering significant inference-time efficiency and robustness to feature and topology perturbations. These properties make E2GNN a practical approach for scalable, reliable GNN ensembles in real-world semi-supervised learning tasks.

Abstract

This work studies ensemble learning for graph neural networks (GNNs) under the popular semi-supervised setting. Ensemble learning has shown superiority in improving the accuracy and robustness of traditional machine learning by combining the outputs of multiple weak learners. However, adopting a similar idea to integrate different GNN models is challenging because of two reasons. First, GNN is notorious for its poor inference ability, so naively assembling multiple GNN models would deteriorate the inference efficiency. Second, when GNN models are trained with few labeled nodes, their performance are limited. In this case, the vanilla ensemble approach, e.g., majority vote, may be sub-optimal since most base models, i.e., GNNs, may make the wrong predictions. To this end, in this paper, we propose an efficient ensemble learner--E2GNN to assemble multiple GNNs in a learnable way by leveraging both labeled and unlabeled nodes. Specifically, we first pre-train different GNN models on a given data scenario according to the labeled nodes. Next, instead of directly combing their outputs for label inference, we train a simple multi-layer perceptron--MLP model to mimic their predictions on both labeled and unlabeled nodes. Then the unified MLP model is deployed to infer labels for unlabeled or new nodes. Since the predictions of unlabeled nodes from different GNN models may be incorrect, we develop a reinforced discriminator to effectively filter out those wrongly predicted nodes to boost the performance of MLP. By doing this, we suggest a principled approach to tackle the inference issues of GNN ensembles and maintain the merit of ensemble learning: improved performance. Comprehensive experiments over both transductive and inductive settings, across different GNN backbones and 8 benchmark datasets, demonstrate the superiority of E2GNN.

Figures (7)

• Figure 1: The performance of five popular GNN models including GCN kipf2016semi, GAT velivckovic2017graph, APPNP klicpera2018predict, GraphSAGE hamilton2017inductive, and SGC wu2019simplifying on 9 random nodes. The color represents certainty, i.e., negative entropy; warmer colors indicate higher certainty. Red starts ($\bigstar$) mark the correct prediction. GNNs tend to make wrong predictions with high certainty.
• Figure 2: The flowchart of the proposed E2GNN framework. Instead of averaging the output of GNN models, E2GNN compresses multiple GNN models into an inference-friendly MLP model via a node-level agent. The agent enables us to adaptively choose the correctly predicted nodes and filter out those incorrectly predicted by all GNN models (i.e., null action) for distillation.
• Figure 3: The performance of five popular GNN models including GCN kipf2016semi, GAT velivckovic2017graph, APPNP klicpera2018predict, GraphSAGE hamilton2017inductive, and SGC wu2019simplifying. The x-axis indicates the number of GNN models that make the correct predictions on some nodes. The y-axis is the ratio of these nodes over all nodes in the graph.
• Figure 4: Visualization of the meta-policy's decisions. The x-axis indicates six groups of nodes, where the value means the number of teacher GNNs that can correctly predict the nodes in the corresponding group. The green bar represents the ratio of the nodes in a group among all the nodes. The orange bar indicates the ratio that the meta-policy makes a good decision (i.e., the selected teacher makes correct predictions for groups 1-5 or take the "null" action for group 0). $\Delta_{*}$ indicates the performance gap between GNN models and E2GNN under different groups, which is the lower the bette. For example, $\Delta_2$ means the ratio of nodes which are correctly predicted by 2 GNN models yet filtered out by E2GNN. Obviously, E2GNN can sample nodes that are correctly predicted by at least one GNN model while rejecting nodes being wrongly classified by all GNNs, i.e., group "0".
• Figure 5: Robust analysis of E2GNN on feature masking.