Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Subgraph-level Universal Prompt Tuning

Junhyun Lee, Wooseong Yang, Jaewoo Kang

TL;DR

In SUPT, prompt features are assigned at the subgraph-level, preserving the method's universal capability, and it excels in few-shot scenarios, achieving an average performance increase of more than 6.6%.

Abstract

In the evolving landscape of machine learning, the adaptation of pre-trained models through prompt tuning has become increasingly prominent. This trend is particularly observable in the graph domain, where diverse pre-training strategies present unique challenges in developing effective prompt-based tuning methods for graph neural networks. Previous approaches have been limited, focusing on specialized prompting functions tailored to models with edge prediction pre-training tasks. These methods, however, suffer from a lack of generalizability across different pre-training strategies. Recently, a simple prompt tuning method has been designed for any pre-training strategy, functioning within the input graph's feature space. This allows it to theoretically emulate any type of prompting function, thereby significantly increasing its versatility for a range of downstream applications. Nevertheless, the capacity of such simple prompts to fully grasp the complex contexts found in graphs remains an open question, necessitating further investigation. Addressing this challenge, our work introduces the Subgraph-level Universal Prompt Tuning (SUPT) approach, focusing on the detailed context within subgraphs. In SUPT, prompt features are assigned at the subgraph-level, preserving the method's universal capability. This requires extremely fewer tuning parameters than fine-tuning-based methods, outperforming them in 42 out of 45 full-shot scenario experiments with an average improvement of over 2.5%. In few-shot scenarios, it excels in 41 out of 45 experiments, achieving an average performance increase of more than 6.6%.

Subgraph-level Universal Prompt Tuning

TL;DR

In SUPT, prompt features are assigned at the subgraph-level, preserving the method's universal capability, and it excels in few-shot scenarios, achieving an average performance increase of more than 6.6%.

Abstract

In the evolving landscape of machine learning, the adaptation of pre-trained models through prompt tuning has become increasingly prominent. This trend is particularly observable in the graph domain, where diverse pre-training strategies present unique challenges in developing effective prompt-based tuning methods for graph neural networks. Previous approaches have been limited, focusing on specialized prompting functions tailored to models with edge prediction pre-training tasks. These methods, however, suffer from a lack of generalizability across different pre-training strategies. Recently, a simple prompt tuning method has been designed for any pre-training strategy, functioning within the input graph's feature space. This allows it to theoretically emulate any type of prompting function, thereby significantly increasing its versatility for a range of downstream applications. Nevertheless, the capacity of such simple prompts to fully grasp the complex contexts found in graphs remains an open question, necessitating further investigation. Addressing this challenge, our work introduces the Subgraph-level Universal Prompt Tuning (SUPT) approach, focusing on the detailed context within subgraphs. In SUPT, prompt features are assigned at the subgraph-level, preserving the method's universal capability. This requires extremely fewer tuning parameters than fine-tuning-based methods, outperforming them in 42 out of 45 full-shot scenario experiments with an average improvement of over 2.5%. In few-shot scenarios, it excels in 41 out of 45 experiments, achieving an average performance increase of more than 6.6%.
Paper Structure (28 sections, 5 theorems, 15 equations, 2 figures, 6 tables)

This paper contains 28 sections, 5 theorems, 15 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Graph Pre-training.
  4. Graph Prompt Tuning.
  5. Method
  6. Preliminaries
  7. Fine-Tuning.
  8. Prompt Tuning.
  9. Graph Prompt Feature.
  10. Subgraph-level Universal Prompt Tuning
  11. SUPT$_{\text{soft}}$
  12. SUPT$_{\text{hard}}$
  13. Methodological Characteristics of Two Variants
  14. Theoretical Analysis
  15. Experiments
  16. ...and 13 more sections

Key Result

Theorem 1

Let $f$ be a pre-trained GNN model and $\mathcal{G} : (X,A)$ be an input graph with sets of nodes and edges, $\mathcal{V}$ and $\mathcal{E}$. Given any prompting function $\psi_t(\cdot)$, if a prompted graph $\hat{\mathcal{G}} : (\hat{X} \in \mathbb{X}, \hat{A} \in \mathbb{A})$ is in the candidate s such that there exists $X'$ satisfying:

Figures (2)

  • Figure 1: Illustration of the concepts: (a) pixel-level visual prompts bahng2022visual1017139710097245liu2023explicitli2023exploringOh_2023_CVPRtsao2024autovp and (b) node-level graph prompts fang2023universal.
  • Figure 2: Illustration of Different Tuning Approaches: Comparing Fine-Tuning, GPF, GPF-Plus, and SUPT. GPF assigns a uniform prompt feature across all nodes, irrespective of their contextual disparities. GPF-Plus advances by allocating prompt features according to node type with category-specific uniformity but not individual graph contextuality. In contrast, SUPT innovates by assigning prompt features at the subgraph level, capturing the intricate contextual nuances of each subgraph. This distinction allows SUPT to apply varied prompt features to nodes of the same type, contingent on their subgraph context, thus introducing a layer of context-specific differentiation even among nodes sharing the same type.

Theorems & Definitions (5)

  • Theorem 1: Universal Capability of SUPT
  • Theorem 1: Universal Capability of SUPT
  • Proposition 1
  • Proposition 2
  • Proposition 3