Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

TL;DR

The paper tackles prompt-tuning for text classification by addressing verbalizer bias and limited coverage through Knowledgeable Prompt-tuning (KPT). KPT expands the verbalizer with external knowledge bases, refines the expanded label-word set via four strategies, and utilizes the refined words using either simple or weighted aggregation to map word-level predictions to class labels. Empirical results on zero-shot and few-shot settings across multiple datasets show that KPT improves accuracy and stabilizes predictions compared to standard prompt-tuning and other baselines, with larger gains in topic classification tasks. The work demonstrates that incorporating structured external knowledge into verbalizers yields more comprehensive, less biased label representations and can reduce variance in data-scarce regimes, suggesting promising directions for knowledge-grounded prompting beyond classification.

Abstract

Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT), to improve and stabilize prompt-tuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.

Figures (4)

• Figure 1: The illustration of KPT , the knowledgeable verbalizer maps the predictions over label words into labels. And the above part is the construction, refinement and utilization processes of KPT .
• Figure 2: Frequent words appearing in the top-5 predictions. The results for two classes: SPORTS (left) and BUSINESS (right) are drawn.
• Figure 3: Support set size w.r.t. zero-shot performance. The points at size=0 are the performances of $\text{PMI}_\text{DC}$.
• Figure 4: The number of remaining label words after Frequency Refinement and Relevance Refinement.