Graph Distillation with Eigenbasis Matching

TL;DR

Graph Distillation with Eigenbasis Matching (GDEM) tackles spectrum bias and cross-architecture inefficiency in graph distillation by aligning the eigenbasis and node features between real and synthetic graphs and directly copying the real spectrum. It introduces a two-stage approach: eigenbasis matching with regularizers to preserve spectral structure, and spectrum-aware reconstruction aided by a discrimination constraint that preserves class-level information, underpinned by a theoretical restricted spectral similarity (RSS) guarantee $ (1-\epsilon) x^T L x < x'^T L' x' < (1+\epsilon) x^T L x $. Empirically, GDEM achieves state-of-the-art or competitive node classification performance across seven datasets, while exhibiting superior cross-architecture generalization and reduced distillation time compared to gradient- or trajectory-based GD methods. The method removes the need to traverse multiple GNN architectures during distillation, enabling more scalable and efficient graph-learning pipelines while maintaining task performance on downstream analyzes.

Abstract

The increasing amount of graph data places requirements on the efficient training of graph neural networks (GNNs). The emerging graph distillation (GD) tackles this challenge by distilling a small synthetic graph to replace the real large graph, ensuring GNNs trained on real and synthetic graphs exhibit comparable performance. However, existing methods rely on GNN-related information as supervision, including gradients, representations, and trajectories, which have two limitations. First, GNNs can affect the spectrum (i.e., eigenvalues) of the real graph, causing spectrum bias in the synthetic graph. Second, the variety of GNN architectures leads to the creation of different synthetic graphs, requiring traversal to obtain optimal performance. To tackle these issues, we propose Graph Distillation with Eigenbasis Matching (GDEM), which aligns the eigenbasis and node features of real and synthetic graphs. Meanwhile, it directly replicates the spectrum of the real graph and thus prevents the influence of GNNs. Moreover, we design a discrimination constraint to balance the effectiveness and generalization of GDEM. Theoretically, the synthetic graphs distilled by GDEM are restricted spectral approximations of the real graphs. Extensive experiments demonstrate that GDEM outperforms state-of-the-art GD methods with powerful cross-architecture generalization ability and significant distillation efficiency. Our code is available at https://github.com/liuyang-tian/GDEM.

Figures (5)

• Figure 1: Data distribution of the real and synthetic graphs in Pubmed dataset, where the average TV of the real graph is 0.87. Left: Synthetic graph distilled by a low-pass filter has a lower value of TV (0.75). Right: Synthetic graph distilled by a high-pass filter has a higher value of TV (1.02). For clarity, only the first 100-dimensional features are visualized. Best viewed in color.
• Figure 2: Comparison between different graph distillation methods, where the red characters represent the synthetic data, the solid black lines, and red dotted lines indicate the forward and backward passes, respectively.
• Figure 3: Influence of $\mathcal{L}_e$ and $\mathcal{L}_d$ in GDEM.
• Figure 4: TVs of synthetic graphs distilled by different methods.
• Figure 5: TVs of synthetic graphs at different epochs (GDEM).

Theorems & Definitions (8)

• Lemma 3.1
• proof
• Lemma 3.2
• proof
• Definition 5.1
• Definition 5.2
• Proposition 5.3
• proof