Manifold-based Verbalizer Space Re-embedding for Tuning-free Prompt-based Classification

TL;DR

This work tackles the tuning burden in prompt-based classification by recognizing that verbalizer embeddings often lie on non-linear manifolds, where Euclidean distance is misleading. It introduces Locally Linear Embedding with Intra-class Neighborhood Constraint (LLE-INC), a tuning-free method that re-embeds the verbalizer space to preserve intra-class locality, followed by kNN classification in the re-embedded space. A contrasting updating module using InfoNCE further enhances PTM representations when tuning is allowed. Across 10 benchmarks and large language models like LLaMA, LLE-INC demonstrates competitive or superior performance without tuning, and gains when combined with parameter updates, underscoring its practical potential for resource-efficient classification on hyper-scale models.

Abstract

Prompt-based classification adapts tasks to a cloze question format utilizing the [MASK] token and the filled tokens are then mapped to labels through pre-defined verbalizers. Recent studies have explored the use of verbalizer embeddings to reduce labor in this process. However, all existing studies require a tuning process for either the pre-trained models or additional trainable embeddings. Meanwhile, the distance between high-dimensional verbalizer embeddings should not be measured by Euclidean distance due to the potential for non-linear manifolds in the representation space. In this study, we propose a tuning-free manifold-based space re-embedding method called Locally Linear Embedding with Intra-class Neighborhood Constraint (LLE-INC) for verbalizer embeddings, which preserves local properties within the same class as guidance for classification. Experimental results indicate that even without tuning any parameters, our LLE-INC is on par with automated verbalizers with parameter tuning. And with the parameter updating, our approach further enhances prompt-based tuning by up to 3.2%. Furthermore, experiments with the LLaMA-7B&13B indicate that LLE-INC is an efficient tuning-free classification approach for the hyper-scale language models.

Figures (3)

• Figure 1: Illustration of three typical verbalizer design methods for prompt-based tuning.
• Figure 2: A sketch map for the space re-embedding. $d_{euc}$ is the Euclidean distance in the original space. $d_{manifold}$ is the distance along the manifold shape in the original space. $d_{re\_embed}$ is the Euclidean distance in the re-embedded space.
• Figure 3: Manifold-based verbalizer space re-embedding for prompt-based tuning. LLE-INC re-embeds the representation space based on the intra-neighbor constraint within the training instances and a kNN classifier makes predictions with the re-embedded representation. The contrastive learning (②$^{*}$) is a supplementary module and is not essential.