Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

TL;DR

This work tackles the latency barrier of graph neural networks by distilling graph-aware knowledge from a GNN teacher into a graph-less MLP student, yielding Graph-less Neural Networks (GLNNs). Through offline knowledge distillation, GLNNs inherit graph-context during training but require no graph data during inference, achieving 146×–273× faster inference than GNNs and 14×–27× faster than other accelerations. Empirically, GLNNs close or match GNN performance on most transductive and production-like inductive settings across seven datasets, with notable benefits arising from larger MLP capacity and informative soft labels. The approach offers a practical pathway for latency-constrained deployment of graph-structured learning while maintaining strong predictive power.

Abstract

Graph Neural Networks (GNNs) are popular for graph machine learning and have shown great results on wide node classification tasks. Yet, they are less popular for practical deployments in the industry owing to their scalability challenges incurred by data dependency. Namely, GNN inference depends on neighbor nodes multiple hops away from the target, and fetching them burdens latency-constrained applications. Existing inference acceleration methods like pruning and quantization can speed up GNNs by reducing Multiplication-and-ACcumulation (MAC) operations, but the improvements are limited given the data dependency is not resolved. Conversely, multi-layer perceptrons (MLPs) have no graph dependency and infer much faster than GNNs, even though they are less accurate than GNNs for node classification in general. Motivated by these complementary strengths and weaknesses, we bring GNNs and MLPs together via knowledge distillation (KD). Our work shows that the performance of MLPs can be improved by large margins with GNN KD. We call the distilled MLPs Graph-less Neural Networks (GLNNs) as they have no inference graph dependency. We show that GLNNs with competitive accuracy infer faster than GNNs by 146X-273X and faster than other acceleration methods by 14X-27X. Under a production setting involving both transductive and inductive predictions across 7 datasets, GLNN accuracies improve over stand-alone MLPs by 12.36% on average and match GNNs on 6/7 datasets. Comprehensive analysis shows when and why GLNNs can achieve competitive accuracies to GNNs and suggests GLNN as a handy choice for latency-constrained applications.

Figures (8)

• Figure 1: The number of fetches and the inference time of GNNs are both magnitudes more than MLPs and grow exponentially as functions of the number of layers. Left: neighbors need to be fetched for two GNN layers. Middle: the total number of fetches for inference. Right: the total inference time. (Inductive inference for 10 random nodes on OGB Productsogb)
• Figure 2: The GLNN framework: In offline training, a trained GNN teacher is applied on the graph for soft targets. Then, a student MLP is trained on node features guided by the soft targets. The distilled MLP, now GLNN, is deployed for online predictions. Since graph dependency is eliminated for inference, GLNNs infer much faster than GNNs, and hence the name "Graph-less Neural Network."
• Figure 3: Enlarged MLPs ( GLNNs) can match GNN accuracy, but infer dramatically faster. Plots are under the same setting as Figure \ref{['figure:time']}. Left: inference time of MLPs vs. GNN (SAGE) for different model sizes. Right: model accuracy vs. inference time. Note: time axes are log-scaled.
• Figure 4: Left: Node feature noise. GLNN has comparable performance to GNNs only when nodes are less noisy. Adding more noise decreases GLNN performance faster than GNNs. Middle: Inductive split rate. Altering the inductive:transductive ratio in the production setting doesn't affect the accuracy much. Right: Teacher GNN architecture. GLNNs can learn from different GNN teachers to improve over MLPs and achieve comparable results. Accuracies are averaged over five CPF datasets.
• Figure 5: The transductive setting and inductive setting illustrated by a 2-layer GNN. The middle shows the original graph used for training. The left shows the transductive setting, where the test node is in red and within the graph. The right shows the inductive setting, where the test node is an unseen new node.

Theorems & Definitions (4)

• Definition 1: Rooted Graph
• Proposition 1
• Definition 2: Rooted Aggregation Tree
• proof