Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphs

Xingtong Yu, Zhenghao Liu, Yuan Fang, Zemin Liu, Sihong Chen, Xinming Zhang

TL;DR

GraphPrompt is proposed, a novel pre-training and prompting framework on graphs that not only unifies pre-training and downstream tasks into a common task template, but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-trained model in a task-specific manner.

Abstract

Graph neural networks have emerged as a powerful tool for graph representation learning, but their performance heavily relies on abundant task-specific supervision. To reduce labeling requirement, the "pre-train, prompt" paradigms have become increasingly common. However, existing study of prompting on graphs is limited, lacking a universal treatment to appeal to different downstream tasks. In this paper, we propose GraphPrompt, a novel pre-training and prompting framework on graphs. GraphPrompt not only unifies pre-training and downstream tasks into a common task template but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-trained model in a task-specific manner. To further enhance GraphPrompt in these two stages, we extend it into GraphPrompt+ with two major enhancements. First, we generalize several popular graph pre-training tasks beyond simple link prediction to broaden the compatibility with our task template. Second, we propose a more generalized prompt design that incorporates a series of prompt vectors within every layer of the pre-trained graph encoder, in order to capitalize on the hierarchical information across different layers beyond just the readout layer. Finally, we conduct extensive experiments on five public datasets to evaluate and analyze GraphPrompt and GraphPrompt+.

Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphs

TL;DR

GraphPrompt is proposed, a novel pre-training and prompting framework on graphs that not only unifies pre-training and downstream tasks into a common task template, but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-trained model in a task-specific manner.

Abstract

Graph neural networks have emerged as a powerful tool for graph representation learning, but their performance heavily relies on abundant task-specific supervision. To reduce labeling requirement, the "pre-train, prompt" paradigms have become increasingly common. However, existing study of prompting on graphs is limited, lacking a universal treatment to appeal to different downstream tasks. In this paper, we propose GraphPrompt, a novel pre-training and prompting framework on graphs. GraphPrompt not only unifies pre-training and downstream tasks into a common task template but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-trained model in a task-specific manner. To further enhance GraphPrompt in these two stages, we extend it into GraphPrompt+ with two major enhancements. First, we generalize several popular graph pre-training tasks beyond simple link prediction to broaden the compatibility with our task template. Second, we propose a more generalized prompt design that incorporates a series of prompt vectors within every layer of the pre-trained graph encoder, in order to capitalize on the hierarchical information across different layers beyond just the readout layer. Finally, we conduct extensive experiments on five public datasets to evaluate and analyze GraphPrompt and GraphPrompt+.
Paper Structure (20 sections, 19 equations, 5 figures, 8 tables, 1 algorithm)

This paper contains 20 sections, 19 equations, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Problem Definition
  5. Graph Neural Networks
  6. Proposed Approach: GraphPrompt
  7. Unification Framework
  8. Pre-Training Phase
  9. Prompting for Downstream Tasks
  10. Extension to GraphPrompt+
  11. Generalizing Pre-Training Tasks
  12. Generalizing Prompts
  13. Experiments
  14. Experimental Setup
  15. Performance Evaluation
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Overall framework of GraphPrompt.
  • Figure 2: Layer-wise prompt tuning for a pre-trained graph encoder with $L$ layers. GraphEncoder$^l$ represents the $l$-th layer of the graph encoder, and $\Vec{p}^{l}$ represents the prompt vector that modifies the $l$-th layer.
  • Figure 3: Impact of shots on few-shot node classification.
  • Figure 4: Impact of shots on few-shot graph classification.
  • Figure 5: Ablation study.