All You Need is One: Capsule Prompt Tuning with a Single Vector

TL;DR

The paper tackles inefficiencies in prompt-based PEFT, notably the reliance on laborious prompt-length searches and the limited interaction of task-aware prompts with input sequences. It introduces Capsule Prompt-Tuning (CaPT), which injects instance-aware semantics through a single per-layer capsule vector to form a concise capsule prompt, combined with a compact task-aware guidance, achieving near parameter-free fine-tuning. Empirically, CaPT delivers superior or on-par performance across multiple backbones on six SuperGLUE tasks while using an imperceptible fraction of parameters and without grid-search, and it exhibits an attention anchor effect that strengthens grounding of input structure. The work suggests a practical, scalable path for efficient LLM adaptation with robust attention dynamics, and points to broader applicability and extensions to other modalities.

Abstract

Prompt-based learning has emerged as a parameter-efficient finetuning (PEFT) approach to facilitate Large Language Model (LLM) adaptation to downstream tasks by conditioning generation with task-aware guidance. Despite its successes, current prompt-based learning methods heavily rely on laborious grid searching for optimal prompt length and typically require considerable number of prompts, introducing additional computational burden. Worse yet, our pioneer findings indicate that the task-aware prompt design is inherently limited by its absence of instance-aware information, leading to a subtle attention interplay with the input sequence. In contrast, simply incorporating instance-aware information as a part of the guidance can enhance the prompt-tuned model performance without additional fine-tuning. Moreover, we find an interesting phenomenon, namely "attention anchor", that incorporating instance-aware tokens at the earliest position of the sequence can successfully preserve strong attention to critical structural information and exhibit more active attention interaction with all input tokens. In light of our observation, we introduce Capsule Prompt-Tuning (CaPT), an efficient and effective solution that leverages off-the-shelf, informative instance semantics into prompt-based learning. Our approach innovatively integrates both instance-aware and task-aware information in a nearly parameter-free manner (i.e., one single capsule prompt). Empirical results demonstrate that our method can exhibit superior performance across various language tasks (e.g., 84.03\% average accuracy on T5-Large), serving as an "attention anchor," while enjoying high parameter efficiency (e.g., 0.003\% of model parameters on Llama3.2-1B).

Figures (10)

• Figure 1: Attention Analysis on T5-Base. Attention patterns are analyzed averagely across all heads and encoder layers on the RTE validation set. Darker red indicates higher attention scores, while darker blue means lower scores. The left and right figures indicate the patterns of T5-Base with 10 soft prompts and T5-Base with one additional prepended instance-aware token (i.e., ★) alongside 10 soft prompts, respectively. Critical input tokens refer to structurally important tokens such as the token "_" at $1_{st}$ and the $5_{th}$ - $7_{th}$ tokens "sentence", "1" and ":". A more detailed analysis of impact on specific heads is provided in Appendix §\ref{['appendix:case_study']}.
• Figure 2: Incorporating of instance-aware semantics lifts the model performance during testing without any additional fine-tuning. (0 token indicates the original accuracy with standard soft prompts only.)
• Figure 3: Overview of Deep Prompt-Tuning $vs.$ CaPT (ours) Frameworks. (a) Original Deep Prompt-Tuning. (b) The overall architecture of our proposed CaPT (see §\ref{['CaPT']}), integrating both task-aware guidance and instance-aware signal to trigger "attention anchor" (see §\ref{['sec:our_attention']}).
• Figure 4: Attention Analysis on T5-Base. Attention patterns are analyzed averagely across all heads and encoder layers. Per layer encoder attention analysis and decoder attention (i.e., causal attention) analysis are shown in Appendix §\ref{['appendix:per_layer_attn']} and §\ref{['appendix:llama_attn']}, respectively.
• Figure 5: CaPT Variants. Three additional designs of CaPT are illustrated: Prepending, Extraction and Projection. The latter two designs require additional trainable modules to process hidden state (e.g., 1D CNN, low-rank Linear). Of all the variants, our default addition design offers the best balance of efficiency and effectiveness, see Table \ref{['table:variants']}.