Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs

Xingtong Yu, Chang Zhou, Yuan Fang, Xinming Zhang

TL;DR

MultiGPrompt presents a novel framework for graph representation learning that integrates multi-task self-supervised pre-training with a dual-prompting strategy to aid few-shot downstream tasks. By introducing per-task pretext tokens and a composed-plus-open prompt design, it achieves synergistic knowledge acquisition across diverse pretext tasks and leverages both task-specific and global pre-trained information during adaptation. Across six public datasets and multiple few-shot settings, the approach consistently outperforms baselines and demonstrates strong cross-dataset transfer, while maintaining high parameter efficiency. This work advances graph pre-training by enabling richer, more transferable representations with minimal downstream tuning, enabling robust performance in label-scarce scenarios.

Abstract

Graphs can inherently model interconnected objects on the Web, thereby facilitating a series of Web applications, such as web analyzing and content recommendation. Recently, Graph Neural Networks (GNNs) have emerged as a mainstream technique for graph representation learning. However, their efficacy within an end-to-end supervised framework is significantly tied to the availabilityof task-specific labels. To mitigate labeling costs and enhance robustness in few-shot settings, pre-training on self-supervised tasks has emerged as a promising method, while prompting has been proposed to further narrow the objective gap between pretext and downstream tasks. Although there has been some initial exploration of prompt-based learning on graphs, they primarily leverage a single pretext task, resulting in a limited subset of general knowledge that could be learned from the pre-training data. Hence, in this paper, we propose MultiGPrompt, a novel multi-task pre-training and prompting framework to exploit multiple pretext tasks for more comprehensive pre-trained knowledge. First, in pre-training, we design a set of pretext tokens to synergize multiple pretext tasks. Second, we propose a dual-prompt mechanism consisting of composed and open prompts to leverage task-specific and global pre-training knowledge, to guide downstream tasks in few-shot settings. Finally, we conduct extensive experiments on six public datasets to evaluate and analyze MultiGPrompt.

MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs

TL;DR

MultiGPrompt presents a novel framework for graph representation learning that integrates multi-task self-supervised pre-training with a dual-prompting strategy to aid few-shot downstream tasks. By introducing per-task pretext tokens and a composed-plus-open prompt design, it achieves synergistic knowledge acquisition across diverse pretext tasks and leverages both task-specific and global pre-trained information during adaptation. Across six public datasets and multiple few-shot settings, the approach consistently outperforms baselines and demonstrates strong cross-dataset transfer, while maintaining high parameter efficiency. This work advances graph pre-training by enabling richer, more transferable representations with minimal downstream tuning, enabling robust performance in label-scarce scenarios.

Abstract

Graphs can inherently model interconnected objects on the Web, thereby facilitating a series of Web applications, such as web analyzing and content recommendation. Recently, Graph Neural Networks (GNNs) have emerged as a mainstream technique for graph representation learning. However, their efficacy within an end-to-end supervised framework is significantly tied to the availabilityof task-specific labels. To mitigate labeling costs and enhance robustness in few-shot settings, pre-training on self-supervised tasks has emerged as a promising method, while prompting has been proposed to further narrow the objective gap between pretext and downstream tasks. Although there has been some initial exploration of prompt-based learning on graphs, they primarily leverage a single pretext task, resulting in a limited subset of general knowledge that could be learned from the pre-training data. Hence, in this paper, we propose MultiGPrompt, a novel multi-task pre-training and prompting framework to exploit multiple pretext tasks for more comprehensive pre-trained knowledge. First, in pre-training, we design a set of pretext tokens to synergize multiple pretext tasks. Second, we propose a dual-prompt mechanism consisting of composed and open prompts to leverage task-specific and global pre-training knowledge, to guide downstream tasks in few-shot settings. Finally, we conduct extensive experiments on six public datasets to evaluate and analyze MultiGPrompt.
Paper Structure (19 sections, 17 equations, 6 figures, 5 tables)

This paper contains 19 sections, 17 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Proposed Approach
  5. Overall Framework
  6. Multi-task Pre-training
  7. Prompting for Downstream Tasks
  8. Experiments
  9. Experimental Setup
  10. Few-shot Performance Evaluation
  11. Ablation study
  12. Further Model Analysis
  13. Conclusions
  14. Further Descriptions of Datasets
  15. Further Descriptions of Baselines
  16. ...and 4 more sections

Figures (6)

  • Figure 1: Illustration of MultiGPrompt. (a) Multi-task pre-training on graphs. (b) Prompting on downstream tasks.
  • Figure 2: Overall framework of MultiGPrompt, consisting of two main stages: (a) Multi-task pre-training, and (b)/(c) Prompt-based learning for downstream few-shot tasks.
  • Figure 3: Application of pretext tokens to the graph encoder. $\Vec{t}_{\langle k\rangle,l}$ represents the pretext token that modifies the $l$-th layer of the graph encoder for the $k$-th pretext task.
  • Figure 4: Impact of shots on node and graph classification.
  • Figure 5: Ablation study on pretext tasks.
  • ...and 1 more figures