Pre-Training and Prompting for Few-Shot Node Classification on Text-Attributed Graphs

TL;DR

This paper addresses few-shot node classification on text-attributed graphs by introducing P2TAG, a framework that jointly pre-trains a language model and a graph neural network using a masked language modeling objective and GraphSAINT-based mini-batches, followed by a mixed graph-text prompting phase. The graph-text prompts align the pre-trained model with downstream tasks by initializing a prompt graph from label texts and combining LM outputs with trainable graph embeddings. Empirical results across six TAG datasets show substantial gains over meta-learning and prior pre-training methods, highlighting the approach's effectiveness and practicality. The work advances scalable, label-efficient graph learning by integrating textual and structural signals in a unified pre-training and prompting paradigm, with public code released for reproducibility and further research impact.

Abstract

The text-attributed graph (TAG) is one kind of important real-world graph-structured data with each node associated with raw texts. For TAGs, traditional few-shot node classification methods directly conduct training on the pre-processed node features and do not consider the raw texts. The performance is highly dependent on the choice of the feature pre-processing method. In this paper, we propose P2TAG, a framework designed for few-shot node classification on TAGs with graph pre-training and prompting. P2TAG first pre-trains the language model (LM) and graph neural network (GNN) on TAGs with self-supervised loss. To fully utilize the ability of language models, we adapt the masked language modeling objective for our framework. The pre-trained model is then used for the few-shot node classification with a mixed prompt method, which simultaneously considers both text and graph information. We conduct experiments on six real-world TAGs, including paper citation networks and product co-purchasing networks. Experimental results demonstrate that our proposed framework outperforms existing graph few-shot learning methods on these datasets with +18.98% ~ +35.98% improvements.

Figures (3)

• Figure 1: Our proposed framework P2TAG. A toy example illustrating the 3-way classification of children's books. (1) In the pre-training phase, we jointly train the LM and GNN using a masked language modeling objective. For the GNN module, a subgraph-based sampler employing random walks generates mini-batch subgraphs for training. (2) In the prompting phase, we construct the ego graph for each target node and generate node features with the pre-trained LM. Graph tokens are utilized to learn the graph structure and create a prompt graph. These tokens are initialized either by encoding label text or through a random initialization process.
• Figure 2: Initialize tokens via the label text. Crafting the prompt graph via downstream label texts under the homophily hypothesis, thereby making the target node embedding more aligned with the downstream label space.
• Figure 3: Effect of different prompt types. We omit label initialization on the prompt graph (w/o $P_{G}$) and node text embedding (w/o $\mathbf{w}_{t}$) and report the classification results on two datasets.

Theorems & Definitions (1)

• Definition 1: Text-Attributed Graph