Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inductive Graph Alignment Prompt: Bridging the Gap between Graph Pre-training and Inductive Fine-tuning From Spectral Perspective

Yuchen Yan, Peiyan Zhang, Zheng Fang, Qingqing Long

TL;DR

This work tackles the challenge of transferring knowledge from graph pre-training to inductive fine-tuning, where pre-training and fine-tuning graphs may differ substantially. It introduces Inductive Graph Alignment Prompt (IGAP), a spectral-space prompt framework that explicitly addresses two data gaps: graph signal gap and graph structure gap, using a learnable graph signal prompt and a recessive spectral-space alignment prompt, plus a label-prompt to align task types. The authors provide a theoretical argument that graph pre-training concentrates on low-frequency spectral components, motivating the alignment of the K smallest spectral modes, and they validate IGAP across node and graph classification tasks in transductive, semi-inductive, and inductive settings. Empirical results show IGAP outperforms or matches strong baselines, with ablations confirming the importance of spectral-space alignment and prompt-based design for effective inductive transfer without access to pre-training data.

Abstract

The "Graph pre-training and fine-tuning" paradigm has significantly improved Graph Neural Networks(GNNs) by capturing general knowledge without manual annotations for downstream tasks. However, due to the immense gap of data and tasks between the pre-training and fine-tuning stages, the model performance is still limited. Inspired by prompt fine-tuning in Natural Language Processing(NLP), many endeavors have been made to bridge the gap in graph domain. But existing methods simply reformulate the form of fine-tuning tasks to the pre-training ones. With the premise that the pre-training graphs are compatible with the fine-tuning ones, these methods typically operate in transductive setting. In order to generalize graph pre-training to inductive scenario where the fine-tuning graphs might significantly differ from pre-training ones, we propose a novel graph prompt based method called Inductive Graph Alignment Prompt(IGAP). Firstly, we unify the mainstream graph pre-training frameworks and analyze the essence of graph pre-training from graph spectral theory. Then we identify the two sources of the data gap in inductive setting: (i) graph signal gap and (ii) graph structure gap. Based on the insight of graph pre-training, we propose to bridge the graph signal gap and the graph structure gap with learnable prompts in the spectral space. A theoretical analysis ensures the effectiveness of our method. At last, we conduct extensive experiments among nodes classification and graph classification tasks under the transductive, semi-inductive and inductive settings. The results demonstrate that our proposed method can successfully bridge the data gap under different settings.

Inductive Graph Alignment Prompt: Bridging the Gap between Graph Pre-training and Inductive Fine-tuning From Spectral Perspective

TL;DR

This work tackles the challenge of transferring knowledge from graph pre-training to inductive fine-tuning, where pre-training and fine-tuning graphs may differ substantially. It introduces Inductive Graph Alignment Prompt (IGAP), a spectral-space prompt framework that explicitly addresses two data gaps: graph signal gap and graph structure gap, using a learnable graph signal prompt and a recessive spectral-space alignment prompt, plus a label-prompt to align task types. The authors provide a theoretical argument that graph pre-training concentrates on low-frequency spectral components, motivating the alignment of the K smallest spectral modes, and they validate IGAP across node and graph classification tasks in transductive, semi-inductive, and inductive settings. Empirical results show IGAP outperforms or matches strong baselines, with ablations confirming the importance of spectral-space alignment and prompt-based design for effective inductive transfer without access to pre-training data.

Abstract

The "Graph pre-training and fine-tuning" paradigm has significantly improved Graph Neural Networks(GNNs) by capturing general knowledge without manual annotations for downstream tasks. However, due to the immense gap of data and tasks between the pre-training and fine-tuning stages, the model performance is still limited. Inspired by prompt fine-tuning in Natural Language Processing(NLP), many endeavors have been made to bridge the gap in graph domain. But existing methods simply reformulate the form of fine-tuning tasks to the pre-training ones. With the premise that the pre-training graphs are compatible with the fine-tuning ones, these methods typically operate in transductive setting. In order to generalize graph pre-training to inductive scenario where the fine-tuning graphs might significantly differ from pre-training ones, we propose a novel graph prompt based method called Inductive Graph Alignment Prompt(IGAP). Firstly, we unify the mainstream graph pre-training frameworks and analyze the essence of graph pre-training from graph spectral theory. Then we identify the two sources of the data gap in inductive setting: (i) graph signal gap and (ii) graph structure gap. Based on the insight of graph pre-training, we propose to bridge the graph signal gap and the graph structure gap with learnable prompts in the spectral space. A theoretical analysis ensures the effectiveness of our method. At last, we conduct extensive experiments among nodes classification and graph classification tasks under the transductive, semi-inductive and inductive settings. The results demonstrate that our proposed method can successfully bridge the data gap under different settings.
Paper Structure (25 sections, 1 theorem, 15 equations, 2 figures, 12 tables)

This paper contains 25 sections, 1 theorem, 15 equations, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Method
  5. Exploring the Essence of Graph Pretraining.
  6. Unifying Pre-training Framework.
  7. The Spectral Character of Graph Pre-training.
  8. Inductive Graph Alignment Prompt
  9. Graph Signal Gap.
  10. Graph Structure Gap.
  11. Task Type Gap.
  12. Experiments
  13. Datasets and Metrics
  14. Baselines
  15. Experimental Settings.
  16. ...and 10 more sections

Key Result

Theorem 1

Graph pre-training aligns the graph signal $x$ more with the low-frequent components than the high-frequent components, where $sim(x,\upsilon_{1})>...>sim(x,\upsilon_{N})$ for $\lambda_{1}<..<\lambda_{N}$.

Figures (2)

  • Figure 1: The framework of IGAP. We first align the graph signals and then we align the spectral space between the pre-train graph and fine-tune graph thus the pre-trained GNN model can be applied. A task-specific prompt is used to align the pre-train task and the fine-tune task.
  • Figure 2: Visualization of Different Baselines on Reddit-F.

Theorems & Definitions (1)

  • Theorem 1