Prompt Tuning on Graph-augmented Low-resource Text Classification

TL;DR

This work tackles low-resource text classification by leveraging the natural graph structure among documents. It introduces Graph-Grounded Pre-training and Prompting (G2P2), a two-stage approach that jointly pre-trains a text encoder and a graph encoder via three graph-grounded contrastive losses and then employs prompting (discrete for zero-shot, continuous for few-shot) to perform classification without updating large PLMs. To handle unseen classes, it extends to G2P2$^*$, which uses a conditional prompting mechanism (Meta-net) to generate input-conditioned prompts for each node, improving generalization. Experiments on four graph-grounded datasets show strong zero- and few-shot performance, with G2P2* delivering robust unseen-class generalization and competitive efficiency due to prompt-based tuning.

Abstract

Text classification is a fundamental problem in information retrieval with many real-world applications, such as predicting the topics of online articles and the categories of e-commerce product descriptions. However, low-resource text classification, with no or few labeled samples, presents a serious concern for supervised learning. Meanwhile, many text data are inherently grounded on a network structure, such as a hyperlink/citation network for online articles, and a user-item purchase network for e-commerce products. These graph structures capture rich semantic relationships, which can potentially augment low-resource text classification. In this paper, we propose a novel model called Graph-Grounded Pre-training and Prompting (G2P2) to address low-resource text classification in a two-pronged approach. During pre-training, we propose three graph interaction-based contrastive strategies to jointly pre-train a graph-text model; during downstream classification, we explore handcrafted discrete prompts and continuous prompt tuning for the jointly pre-trained model to achieve zero- and few-shot classification, respectively. Moreover, we explore the possibility of employing continuous prompt tuning for zero-shot inference. Specifically, we aim to generalize continuous prompts to unseen classes while leveraging a set of base classes. To this end, we extend G2P2 into G2P2$^*$, hinging on a new architecture of conditional prompt tuning. Extensive experiments on four real-world datasets demonstrate the strength of G2P2 in zero- and few-shot low-resource text classification tasks, and illustrate the advantage of G2P2$^*$ in dealing with unseen classes.

Figures (8)

• Figure 1: From G2P2 to G2P2$^*$: Learning generalizable continuous prompts. Illustrations are based on the Amazon Art dataset (see Sect. \ref{['sec:expt:setup']}).
• Figure 2: Overall framework of G2P2. (a) During pre-training, it jointly trains a text and a graph encoder through three contrastive strategies. (b) During testing, it performs prompt-assisted zero- or few-shot classification. Note that part (b) only shows continuous prompt tuning for few-shot classification, while discrete prompts for zero-shot inference and conditional prompt tuning for generalization to wider unseen classes are presented separately in Figs. \ref{['fig:zero-shot']} and \ref{['fig:meta net']}, respectively.
• Figure 3: Schematic diagram for zero-shot classification. The pre-trained models $\theta_G^0$ and $\theta_T^0$ are obtained from Fig. \ref{['fig:framework']}(a).
• Figure 4: Schematic diagram for conditional prompt tuning in G2P2$^*$. The pre-trained models $\theta_G^0$ and $\theta_T^0$ are obtained from Fig. \ref{['fig:framework']}(a). The classes are base class during tuning, and unseen classes during zero-shot inference.
• Figure 5: Classification performance on different shots.