Fast Graph Condensation with Structure-based Neural Tangent Kernel

TL;DR

The paper tackles the inefficiency of graph-data condensation for training GNNs on large graphs by reframing the problem as Kernel Ridge Regression (KRR) instead of a costly bi-level optimization. It introduces GC-SNTK, a framework that uses a Structure-based Neural Tangent Kernel (SNTK) to capture graph topology through neighborhood aggregation within the KRR paradigm, enabling a single-loop optimization. The approach yields substantial speedups over prior bi-level methods while preserving or surpassing predictive performance across multiple graph datasets and GNN architectures, and it demonstrates robust cross-model generalization. These findings suggest a practical route to scalable graph condensation that leverages infinite-width network insights via NTK while respecting graph structure for effective downstream learning.

Abstract

The rapid development of Internet technology has given rise to a vast amount of graph-structured data. Graph Neural Networks (GNNs), as an effective method for various graph mining tasks, incurs substantial computational resource costs when dealing with large-scale graph data. A data-centric manner solution is proposed to condense the large graph dataset into a smaller one without sacrificing the predictive performance of GNNs. However, existing efforts condense graph-structured data through a computational intensive bi-level optimization architecture also suffer from massive computation costs. In this paper, we propose reforming the graph condensation problem as a Kernel Ridge Regression (KRR) task instead of iteratively training GNNs in the inner loop of bi-level optimization. More specifically, We propose a novel dataset condensation framework (GC-SNTK) for graph-structured data, where a Structure-based Neural Tangent Kernel (SNTK) is developed to capture the topology of graph and serves as the kernel function in KRR paradigm. Comprehensive experiments demonstrate the effectiveness of our proposed model in accelerating graph condensation while maintaining high prediction performance. The source code is available on https://github.com/WANGLin0126/GCSNTK.

Figures (6)

• Figure 1: Graph condensation aims to condense graph data to a smaller but informative version. In general, two GNN classifiers (i.e., GNN① and GNN②) are trained on $G^{\mathcal{T}}$ and $G^{\mathcal{S}}$ simultaneously. Meanwhile, various matching objectives on two GNNs are conducted to synthesize $G^{\mathcal{S}}$.
• Figure 2: Bi-level graph condensation optimization (a) and the proposed GC-SNTK (b). $G^{\mathcal{T}}$ and $G^{\mathcal{S}}$ denote the target and condensed graph data. $\mathrm{GNN}_{\theta}$ is the graph neural network model with parameter $\theta$. $\mathcal{L}$ and $\ell$ are the loss of the outer and inner loop, respectively. opt-alg is the optimization algorithm. The bi-level model entails a inner GNN training loop, a outer $G^{\mathcal{S}}$ optmization loop, and $R$-time initialization. On the contrary, the proposed GC-SNTK only have a single $G^{\mathcal{S}}$ optimization loop.
• Figure 3: The comparison of node classification accuracy. (a)-(d) illustrate the performance variation of GC-SNTK as the condensation size decreases to a single node. (e)-(f) represent the performance comparison of GC-SNTK, GCond, and One-Step methods at extremely small condensation sizes.
• Figure 4: Condensation efficiency consumption on the four datasets (the number after the name of dataset is the nodes size of the condensed data).
• Figure 5: Sensitivity analysis of $K$ and $L$ in SNTK. We choose different values of $K$ and $L$ to pose different SNTKs. Their performances are represented as different colors shown above.