Online Adversarial Knowledge Distillation for Graph Neural Networks

TL;DR

This work tackles the challenge that traditional knowledge distillation struggles for Graph Neural Networks when graph topology and node attributes evolve, causing distribution shift between teacher and student. It proposes Online Adversarial Knowledge Distillation (OAD), a group-based online framework that distills local knowledge via adversarial cyclic learning and global knowledge via ensemble peer predictions. The approach yields a lightweight, scalable training scheme with complexity that scales linearly with the number of peers and demonstrates improvements across transductive, inductive, and 3D object recognition tasks, as well as dynamic graphs. The work provides empirical evidence that mutual learning among multiple GNNs can approximate a dynamic virtual teacher, reducing the need for retraining heavy teachers and enabling robust distillation in changing graph environments.

Abstract

Knowledge distillation, a technique recently gaining popularity for enhancing model generalization in Convolutional Neural Networks (CNNs), operates under the assumption that both teacher and student models are trained on identical data distributions. However, its effect on Graph Neural Networks (GNNs) is less than satisfactory since the graph topology and node attributes are prone to evolve, thereby leading to the issue of distribution shift. In this paper, we tackle this challenge by simultaneously training a group of graph neural networks in an online distillation fashion, where the group knowledge plays a role as a dynamic virtual teacher and the structure changes in graph neural networks are effectively captured. To improve the distillation performance, two types of knowledge are transferred among the students to enhance each other: local knowledge reflecting information in the graph topology and node attributes, and global knowledge reflecting the prediction over classes. We transfer the global knowledge with KL-divergence as the vanilla knowledge distillation does, while exploiting the complicated structure of the local knowledge with an efficient adversarial cyclic learning framework. Extensive experiments verified the effectiveness of our proposed online adversarial distillation approach. The code is published at https://github.com/wangz3066/OnlineDistillGCN.

Figures (6)

• Figure 1: Overview of the proposed online adversarial knowledge distillation framework for GNNs. We show the case with four student GNN models. The student network is a GNN generator. Each student model is assigned with a discriminator. The local knowledge is distilled by a cyclic generator-discriminator framework, denoted as red dot lines. The global knowledge is distilled by the KL divergence between student logits and "virtual" teacher's logits, denoted as green dot lines.
• Figure 2: F1 score improvement w.r.t. a single GNN after adding ramdom noise on the node attributions of PPI dataset. The blue lines denote "OAD_Improve" and the orange lines denote "KD_Improve". The left subplot shows the experiment on GAT. The right subplot shows the experiment on GraphSAGE.
• Figure 3: F1 score improvement w.r.t. a single GNN after removing certain proportions of edges of the graphs of PPI dataset. The blue lines denote "OAD_Improve" and the orange lines denote "KD_Improve". The left subplot shows the experiment on GAT. The right subplot shows the experiment on GraphSAGE.
• Figure 4: Visualization of features learned by different models, deprecated by the distance between anchor points (colored in red) and other points. From left to right: original input features; features learned by student DGCNN; features learned by DGCNN with OAD framework and features learned learned by teacher DGCNN. Best viewed in color.
• Figure 5: The training F1 scores of Ensemble, OAD and a vanilla GNN on PPI dataset.