Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation

Lirong Wu, Yunfan Liu, Haitao Lin, Yufei Huang, Stan Z. Li

TL;DR

This work tackles the challenge of transferring knowledge from powerful GNN teachers to lightweight MLP students by focusing on hard samples. It introduces HGMD, a non-parametric, hardness-aware distillation framework that decouples knowledge hardness from distillation hardness, extracts hardness-aware subgraphs per target node, and applies two schemes—HGMD-weight and HGMD-mixup—to distill subgraph-level knowledge. Empirical results across eight real-world datasets show that HGMD variants consistently outperform strong baselines, with HGMD-mixup achieving notable gains on large-scale graphs like ogbn-arxiv, all without introducing extralearnable parameters. The analysis reveals that prioritizing hard samples via structured subgraphs and mixup provides robust supervision, and that the resulting asymmetric subgraph extraction is a key factor in HGMD’s effectiveness.

Abstract

To bridge the gaps between powerful Graph Neural Networks (GNNs) and lightweight Multi-Layer Perceptron (MLPs), GNN-to-MLP Knowledge Distillation (KD) proposes to distill knowledge from a well-trained teacher GNN into a student MLP. In this paper, we revisit the knowledge samples (nodes) in teacher GNNs from the perspective of hardness, and identify that hard sample distillation may be a major performance bottleneck of existing graph KD algorithms. The GNN-to-MLP KD involves two different types of hardness, one student-free knowledge hardness describing the inherent complexity of GNN knowledge, and the other student-dependent distillation hardness describing the difficulty of teacher-to-student distillation. However, most of the existing work focuses on only one of these aspects or regards them as one thing. This paper proposes a simple yet effective Hardness-aware GNN-to-MLP Distillation (HGMD) framework, which decouples the two hardnesses and estimates them using a non-parametric approach. Finally, two hardness-aware distillation schemes (i.e., HGMD-weight and HGMD-mixup) are further proposed to distill hardness-aware knowledge from teacher GNNs into the corresponding nodes of student MLPs. As non-parametric distillation, HGMD does not involve any additional learnable parameters beyond the student MLPs, but it still outperforms most of the state-of-the-art competitors. HGMD-mixup improves over the vanilla MLPs by 12.95% and outperforms its teacher GNNs by 2.48% averaged over seven real-world datasets.

Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation

TL;DR

This work tackles the challenge of transferring knowledge from powerful GNN teachers to lightweight MLP students by focusing on hard samples. It introduces HGMD, a non-parametric, hardness-aware distillation framework that decouples knowledge hardness from distillation hardness, extracts hardness-aware subgraphs per target node, and applies two schemes—HGMD-weight and HGMD-mixup—to distill subgraph-level knowledge. Empirical results across eight real-world datasets show that HGMD variants consistently outperform strong baselines, with HGMD-mixup achieving notable gains on large-scale graphs like ogbn-arxiv, all without introducing extralearnable parameters. The analysis reveals that prioritizing hard samples via structured subgraphs and mixup provides robust supervision, and that the resulting asymmetric subgraph extraction is a key factor in HGMD’s effectiveness.

Abstract

To bridge the gaps between powerful Graph Neural Networks (GNNs) and lightweight Multi-Layer Perceptron (MLPs), GNN-to-MLP Knowledge Distillation (KD) proposes to distill knowledge from a well-trained teacher GNN into a student MLP. In this paper, we revisit the knowledge samples (nodes) in teacher GNNs from the perspective of hardness, and identify that hard sample distillation may be a major performance bottleneck of existing graph KD algorithms. The GNN-to-MLP KD involves two different types of hardness, one student-free knowledge hardness describing the inherent complexity of GNN knowledge, and the other student-dependent distillation hardness describing the difficulty of teacher-to-student distillation. However, most of the existing work focuses on only one of these aspects or regards them as one thing. This paper proposes a simple yet effective Hardness-aware GNN-to-MLP Distillation (HGMD) framework, which decouples the two hardnesses and estimates them using a non-parametric approach. Finally, two hardness-aware distillation schemes (i.e., HGMD-weight and HGMD-mixup) are further proposed to distill hardness-aware knowledge from teacher GNNs into the corresponding nodes of student MLPs. As non-parametric distillation, HGMD does not involve any additional learnable parameters beyond the student MLPs, but it still outperforms most of the state-of-the-art competitors. HGMD-mixup improves over the vanilla MLPs by 12.95% and outperforms its teacher GNNs by 2.48% averaged over seven real-world datasets.
Paper Structure (25 sections, 10 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 25 sections, 10 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. GNN-to-GNN Knowledge Distillation
  4. GNN-to-MLP Knowledge Distillation
  5. Methodology
  6. Preliminary
  7. Notations.
  8. Knowledge Hardness.
  9. Bottleneck: Hard Sample Distillation
  10. Hardness-aware GNN-to-MLP Distillation
  11. Hardness-aware Subgraph Extraction
  12. HGMD-weight
  13. HGMD-mixup
  14. Training Strategy
  15. Parameters and Computational Complexity
  16. ...and 10 more sections

Figures (5)

  • Figure 1: (a) Accuracy rankings of three teacher GNNs (GCN, SAGE, and GAT) and corresponding student MLPs on seven datasets. (b) Accuracy fluctuations of the teacher GCN and student MLP w.r.t temperature coefficient $\tau$ on the Cora dataset. (c) Histogram of the information entropy of GNN knowledge for those samples misclassified by student MLPs on the Cora dataset.
  • Figure 2: Classification accuracy of several representative GNN-to-MLP distillation methods for simple samples (bottom 50% hardness) and hard samples (top 50% hardness).
  • Figure 3: Illustration of the hardness-aware GNN-to-MLP KD (HGMD) framework, which consists of two main components: (1) hardness-aware subgraph extraction; and (2) two hardness-aware distillation schemes (HGMD-weight and HGMD-mixup).
  • Figure 4: Visualizations of three GNN knowledge samples of different hardness levels (Low / Middle / High), where the node and edge colors indicate the hardness of knowledge samples and the sampling probability, and the color bars are on the right.
  • Figure 5: (ab) 3D histogram of the sampling probability of neighboring nodes w.r.t their hardness, hardness of the target node, and their cosine similarity to the target on Cora. (c) Training curves for the average entropy of student MLPs and the average size of sampled subgraphs on Cora. (d) Ratio of connected nodes sampled symmetrically and asymmetrically among all edges.