Universal Prompt Tuning for Graph Neural Networks

TL;DR

This work tackles the challenge of adapting pre-trained GNNs to downstream tasks under diverse pre-training objectives by proposing a universal graph prompt tuning approach. It introduces Graph Prompt Feature (GPF) and its variant GPF-plus, which inject learnable prompts into the input feature space (X) rather than modifying model weights, enabling compatibility with any pre-training strategy. Theoretical results guarantee that GPF is universal with respect to prompting functions and can outperform full fine-tuning in certain regimes, while empirical results show consistent gains over fine-tuning across multiple strategies and datasets, with dramatically fewer tunable parameters. The methods also demonstrate strong performance in few-shot settings and superiority over existing graph prompt methods, offering a practical, scalable alternative for downstream adaptations in graph representation learning.

Abstract

In recent years, prompt tuning has sparked a research surge in adapting pre-trained models. Unlike the unified pre-training strategy employed in the language field, the graph field exhibits diverse pre-training strategies, posing challenges in designing appropriate prompt-based tuning methods for graph neural networks. While some pioneering work has devised specialized prompting functions for models that employ edge prediction as their pre-training tasks, these methods are limited to specific pre-trained GNN models and lack broader applicability. In this paper, we introduce a universal prompt-based tuning method called Graph Prompt Feature (GPF) for pre-trained GNN models under any pre-training strategy. GPF operates on the input graph's feature space and can theoretically achieve an equivalent effect to any form of prompting function. Consequently, we no longer need to illustrate the prompting function corresponding to each pre-training strategy explicitly. Instead, we employ GPF to obtain the prompted graph for the downstream task in an adaptive manner. We provide rigorous derivations to demonstrate the universality of GPF and make guarantee of its effectiveness. The experimental results under various pre-training strategies indicate that our method performs better than fine-tuning, with an average improvement of about 1.4% in full-shot scenarios and about 3.2% in few-shot scenarios. Moreover, our method significantly outperforms existing specialized prompt-based tuning methods when applied to models utilizing the pre-training strategy they specialize in. These numerous advantages position our method as a compelling alternative to fine-tuning for downstream adaptations.

Figures (2)

• Figure 1: Comparison of universal graph prompt tuning and existing approaches.(a) Fine-tuning updates the parameters of the pre-trained GNN model. (b) Existing specialized prompt-based tuning methods generate manual graph templates to adapt the models under certain pre-training strategies. (c) Our universal graph prompt tuning works on the feature space of the input graph. It can achieve an equivalent effect to any form of prompting function and be applied to any pre-trained GNN model.
• Figure 2: Training and test curves of different tuning methods.

Theorems & Definitions (13)

• Theorem 1
• Theorem 2
• Proposition 1
• Proposition 2
• Proposition 3
• proof
• Proposition 4
• proof
• Proposition 5
• proof