InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation
Amir Eskandari, Aman Anand, Elyas Rashno, Farhana Zulkernine
TL;DR
InfGraND tackles the efficiency gap in graph learning by distilling a GNN teacher into an MLP student guided by graph-informed node influence. It computes a graph-centric influence metric, performs a one-time multi-hop feature propagation to enrich inputs, and uses an influence-weighted distillation objective to transfer structural knowledge efficiently. Across seven paralleled datasets and both transductive and inductive settings, InfGraND consistently outperforms prior GNN-to-MLP distillation methods and, in many cases, the teacher itself, while delivering strong latency advantages for deployment. The approach demonstrates practical scalability and opens directions for extending to heterophilic and dynamic graphs, as well as combining structure-aware and uncertainty-based signals in distillation.
Abstract
Graph Neural Networks (GNNs) are the go-to model for graph data analysis. However, GNNs rely on two key operations - aggregation and update, which can pose challenges for low-latency inference tasks or resource-constrained scenarios. Simple Multi-Layer Perceptrons (MLPs) offer a computationally efficient alternative. Yet, training an MLP in a supervised setting often leads to suboptimal performance. Knowledge Distillation (KD) from a GNN teacher to an MLP student has emerged to bridge this gap. However, most KD methods either transfer knowledge uniformly across all nodes or rely on graph-agnostic indicators such as prediction uncertainty. We argue this overlooks a more fundamental, graph-centric inquiry: "How important is a node to the structure of the graph?" We introduce a framework, InfGraND, an Influence-guided Graph KNowledge Distillation from GNN to MLP that addresses this by identifying and prioritizing structurally influential nodes to guide the distillation process, ensuring that the MLP learns from the most critical parts of the graph. Additionally, InfGraND embeds structural awareness in MLPs through one-time multi-hop neighborhood feature pre-computation, which enriches the student MLP's input and thus avoids inference-time overhead. Our rigorous evaluation in transductive and inductive settings across seven homophilic graph benchmark datasets shows InfGraND consistently outperforms prior GNN to MLP KD methods, demonstrating its practicality for numerous latency-critical applications in real-world settings.