Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instance-Aware Graph Prompt Learning

Jiazheng Li, Jundong Li, Chuxu Zhang

TL;DR

Instance-Aware Graph Prompt Learning is introduced, aiming to generate distinct prompts tailored to different input instances, and involves generating intermediate prompts for each instance using a lightweight architecture, quantizing these prompts through trainable codebook vectors, and employing the exponential moving average technique to ensure stable training.

Abstract

Graph neural networks stand as the predominant technique for graph representation learning owing to their strong expressive power, yet the performance highly depends on the availability of high-quality labels in an end-to-end manner. Thus the pretraining and fine-tuning paradigm has been proposed to mitigate the label cost issue. Subsequently, the gap between the pretext tasks and downstream tasks has spurred the development of graph prompt learning which inserts a set of graph prompts into the original graph data with minimal parameters while preserving competitive performance. However, the current exploratory works are still limited since they all concentrate on learning fixed task-specific prompts which may not generalize well across the diverse instances that the task comprises. To tackle this challenge, we introduce Instance-Aware Graph Prompt Learning (IA-GPL) in this paper, aiming to generate distinct prompts tailored to different input instances. The process involves generating intermediate prompts for each instance using a lightweight architecture, quantizing these prompts through trainable codebook vectors, and employing the exponential moving average technique to ensure stable training. Extensive experiments conducted on multiple datasets and settings showcase the superior performance of IA-GPL compared to state-of-the-art baselines.

Instance-Aware Graph Prompt Learning

TL;DR

Instance-Aware Graph Prompt Learning is introduced, aiming to generate distinct prompts tailored to different input instances, and involves generating intermediate prompts for each instance using a lightweight architecture, quantizing these prompts through trainable codebook vectors, and employing the exponential moving average technique to ensure stable training.

Abstract

Graph neural networks stand as the predominant technique for graph representation learning owing to their strong expressive power, yet the performance highly depends on the availability of high-quality labels in an end-to-end manner. Thus the pretraining and fine-tuning paradigm has been proposed to mitigate the label cost issue. Subsequently, the gap between the pretext tasks and downstream tasks has spurred the development of graph prompt learning which inserts a set of graph prompts into the original graph data with minimal parameters while preserving competitive performance. However, the current exploratory works are still limited since they all concentrate on learning fixed task-specific prompts which may not generalize well across the diverse instances that the task comprises. To tackle this challenge, we introduce Instance-Aware Graph Prompt Learning (IA-GPL) in this paper, aiming to generate distinct prompts tailored to different input instances. The process involves generating intermediate prompts for each instance using a lightweight architecture, quantizing these prompts through trainable codebook vectors, and employing the exponential moving average technique to ensure stable training. Extensive experiments conducted on multiple datasets and settings showcase the superior performance of IA-GPL compared to state-of-the-art baselines.

Paper Structure

This paper contains 21 sections, 19 equations, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. METHODOLOGY
  5. Naive Approach
  6. Lightweight Bottleneck Architecture
  7. Prompt Quantization
  8. Model Optimization
  9. Experiments
  10. Experimental Setup
  11. Performance Evaluation
  12. Model Analysis
  13. Broader Impacts and Limitations
  14. Conclusions
  15. Dataset Details
  16. ...and 6 more sections

Figures (7)

  • Figure 1: Two example molecules from the BBBP dataset. Molecule (a) with simple structures suffices with a universal prompt. However, molecule (b) with diverse atoms and intricate structures requires the use of instance-aware prompts.
  • Figure 2: Comparison between different paradigms of graph representation learning.
  • Figure 3: Overall Framework of IA-GPL.
  • Figure 4: Codebook visualization.
  • Figure 5: Ablation study.
  • ...and 2 more figures