Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Token Coordinated Prompt Attention is Needed for Visual Prompting

Zichen Liu, Xu Zou, Gang Hua, Jiahuan Zhou

TL;DR

Existing visual prompting often uses identical prompts for all tokens in Vision Transformers, limiting feature diversity. TCPA introduces CLS prompts for CLS tokens and Image prompts for image tokens, plus a per-token prompt-matching mechanism that diversifies attention interactions while keeping the ViT encoder frozen; only prompts and the classifier head are trained. The approach disentangles prompts by token role, enables per-token prompt assignment for image tokens, and yields consistent improvements across HTA and VTAB benchmarks with modest overhead. Overall, TCPA enhances the diversity and discriminability of visual features, and is designed as a practical, plug-and-play enhancement for ViT-based prompting with broad compatibility.

Abstract

Visual prompting techniques are widely used to efficiently fine-tune pretrained Vision Transformers (ViT) by learning a small set of shared prompts for all tokens. However, existing methods overlook the unique roles of different tokens in conveying discriminative information and interact with all tokens using the same prompts, thereby limiting the representational capacity of ViT. This often leads to indistinguishable and biased prompt-extracted features, hindering performance. To address this issue, we propose a plug-and-play Token Coordinated Prompt Attention (TCPA) module, which assigns specific coordinated prompts to different tokens for attention-based interactions. Firstly, recognizing the distinct functions of CLS and image tokens-global information aggregation and local feature extraction, we disentangle the prompts into CLS Prompts and Image Prompts, which interact exclusively with CLS tokens and image tokens through attention mechanisms. This enhances their respective discriminative abilities. Furthermore, as different image tokens correspond to distinct image patches and contain diverse information, we employ a matching function to automatically assign coordinated prompts to individual tokens. This enables more precise attention interactions, improving the diversity and representational capacity of the extracted features. Extensive experiments across various benchmarks demonstrate that TCPA significantly enhances the diversity and discriminative power of the extracted features. The code is available at https://github.com/zhoujiahuan1991/ICML2025-TCPA.

Token Coordinated Prompt Attention is Needed for Visual Prompting

TL;DR

Existing visual prompting often uses identical prompts for all tokens in Vision Transformers, limiting feature diversity. TCPA introduces CLS prompts for CLS tokens and Image prompts for image tokens, plus a per-token prompt-matching mechanism that diversifies attention interactions while keeping the ViT encoder frozen; only prompts and the classifier head are trained. The approach disentangles prompts by token role, enables per-token prompt assignment for image tokens, and yields consistent improvements across HTA and VTAB benchmarks with modest overhead. Overall, TCPA enhances the diversity and discriminability of visual features, and is designed as a practical, plug-and-play enhancement for ViT-based prompting with broad compatibility.

Abstract

Visual prompting techniques are widely used to efficiently fine-tune pretrained Vision Transformers (ViT) by learning a small set of shared prompts for all tokens. However, existing methods overlook the unique roles of different tokens in conveying discriminative information and interact with all tokens using the same prompts, thereby limiting the representational capacity of ViT. This often leads to indistinguishable and biased prompt-extracted features, hindering performance. To address this issue, we propose a plug-and-play Token Coordinated Prompt Attention (TCPA) module, which assigns specific coordinated prompts to different tokens for attention-based interactions. Firstly, recognizing the distinct functions of CLS and image tokens-global information aggregation and local feature extraction, we disentangle the prompts into CLS Prompts and Image Prompts, which interact exclusively with CLS tokens and image tokens through attention mechanisms. This enhances their respective discriminative abilities. Furthermore, as different image tokens correspond to distinct image patches and contain diverse information, we employ a matching function to automatically assign coordinated prompts to individual tokens. This enables more precise attention interactions, improving the diversity and representational capacity of the extracted features. Extensive experiments across various benchmarks demonstrate that TCPA significantly enhances the diversity and discriminative power of the extracted features. The code is available at https://github.com/zhoujiahuan1991/ICML2025-TCPA.
Paper Structure (23 sections, 2 theorems, 16 equations, 7 figures, 4 tables)

This paper contains 23 sections, 2 theorems, 16 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Parameter-Efficient Fine-Tuning
  4. Prompt Learning
  5. Token Coordinated Prompt Attention
  6. Notations
  7. Coordinated Prompt Attention of CLS and Image Tokens
  8. Coordinated Prompt Attention of Different Image Tokens
  9. Overall Optimization
  10. Discussion and Analysis
  11. Experiments
  12. Datasets
  13. Comparison Methods
  14. Implementation Details
  15. Comparison with State-of-the-arts
  16. ...and 8 more sections

Key Result

Theorem 4.1

Self-attention is low rank. (Proved in wang2020linformer). Let $A\in \mathbb{R}^{n\times n}$ be a self-attention matrix, and $v\in \mathbb{R}^n$ be a column vector of value matrix $V$. Then, there exists a low-rank matrix $\hat{A}\in \mathbb{R}^{n \times n}$ satisfying where the rank of $\hat{A}$ is bounded, i.e., $rank(A)=\Theta(log(n))$.

Figures (7)

  • Figure 1: Above: Visualization of the attention map. The existing visual prompting method VPT jia2022visual learns the same prompts for all tokens, resulting in extracted information that lacks distinguishability and comprehensiveness. Our TCPA selects corresponding prompts for different tokens and performs attention interaction, thereby enhancing the diversity and discriminability of the extracted information. Below: Comparison of time overhead and performance.
  • Figure 2: The overall pipeline of our proposed TCPA. For each input sample, embeddings for each image patch are first obtained through the embedding layer. Then, CLS and image tokens adaptively select appropriate prompts from the corresponding CLS and Image Prompt Pools and generate a binary mask. This binary mask is then fed into the attention module to mask certain values in the attention map, enabling attention-based interactions between different tokens and different prompts.
  • Figure 3: 3D and 2D attention map of existing visual prompting method VPT jia2022visual and Ours.
  • Figure 4: Influence of hyper-parameters (size of CLS prompt pool $N_c$, size of image prompt pool $N_i$) of TCPA on CUB.
  • Figure 5: Feature t-SNE van2008visualizing visualization results for our proposed TCPA and comparison method DAMVP huang2023diversity on GTSRB.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 4.1
  • Theorem 4.2