A Novel Prompt-tuning Method: Incorporating Scenario-specific Concepts into a Verbalizer

TL;DR

The paper introduces ISCV, a verbalizer-construction framework that injects scenario-specific concepts into prompt-tuning to expand label-word coverage and reduce bias. It couples concept mining (via named entity extraction or POS tags and external concept bases) with cascade calibration (anchor creation, language-model calibration, and category calibration) to produce robust label-word sets for zero-shot text classification. Empirical results on five datasets show state-of-the-art zero-shot performance on topic classification and strong results on sentiment tasks, along with enhanced template stability and favorable few-shot behavior. The work demonstrates that leveraging higher-level concepts and task-specific context within verbalizers can substantially improve prompt-based learning, with avenues for automation and multilingual extension future work.

Abstract

The verbalizer, which serves to map label words to class labels, is an essential component of prompt-tuning. In this paper, we present a novel approach to constructing verbalizers. While existing methods for verbalizer construction mainly rely on augmenting and refining sets of synonyms or related words based on class names, this paradigm suffers from a narrow perspective and lack of abstraction, resulting in limited coverage and high bias in the label-word space. To address this issue, we propose a label-word construction process that incorporates scenario-specific concepts. Specifically, we extract rich concepts from task-specific scenarios as label-word candidates and then develop a novel cascade calibration module to refine the candidates into a set of label words for each class. We evaluate the effectiveness of our proposed approach through extensive experiments on {five} widely used datasets for zero-shot text classification. The results demonstrate that our method outperforms existing methods and achieves state-of-the-art results.

Figures (4)

• Figure 1: Illustration of the ISCV motivation. "ExKB" means retrieving words from an external knowledge base. Expansion from multiple perspectives and higher-level abstractions, as well as task-specific refinement, enables large coverage and low bias in a label-word space.
• Figure 2: The architecture of our ISCV including a verbalizer construction approach and a zero-shot learning pipeline. The verbalizer construction approach consists of a concept mining procedure and a cascade calibration procedure. The two light-beige segments constitute the cascade calibration procedure. The section with white color is the concept mining procedure. The bottom part is the zero-shot-based text classification pipeline in prompt-tuning. DS means a dataset.
• Figure 3: Comparison between ISCV and KPT for template stability. The orange color indicates ISCV and the gray color indicates KPT. T1, T2, T3, and T4 denote the four templates for each dataset separately and each number at the top of the colored bar is the value of the corresponding Micro-F1 value.
• Figure 4: Parameter Sensitivity of ISCV on each template. "ssize" denotes the size of a support set.