VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion

TL;DR

VIPAMIN tackles the inefficiency of full fine-tuning by introducing a zero-parameter initialization for visual prompts in self-supervised ViTs. It comprises two modules: a Matching module that aligns prompts to semantically coherent input tokens, and an Orthogonalizing module that injects novel directions outside the pretrained subspace, using a one-time initialization with a single forward pass and two lightweight matrix operations. Across 24 vision tasks including VTAB-1k and FGVC few-shot settings, VIPAMIN consistently improves over standard VPT and other baselines, delivering state-of-the-art performance in many cases, especially under distribution shifts and data scarcity. The method scales to deep VPT and hierarchical backbones, maintains low overhead, and offers interpretable improvements as evidenced by Grad-CAM analyses, indicating better localization of task-relevant regions. Overall, VIPAMIN provides a practical, scalable, and effective approach to prompt-based adaptation for large self-supervised vision models.

Abstract

In the era of large-scale foundation models, fully fine-tuning pretrained networks for each downstream task is often prohibitively resource-intensive. Prompt tuning offers a lightweight alternative by introducing tunable prompts while keeping the backbone frozen. However, existing visual prompt tuning methods often fail to specialize the prompts or enrich the representation space--especially when applied to self-supervised backbones. We show that these limitations become especially pronounced in challenging tasks and data-scarce settings, where effective adaptation is most critical. In this work, we introduce VIPAMIN, a visual prompt initialization strategy that enhances adaptation of self-supervised models by (1) aligning prompts with semantically informative regions in the embedding space, and (2) injecting novel representational directions beyond the pretrained subspace. Despite its simplicity--requiring only a single forward pass and lightweight operations--VIPAMIN consistently improves performance across diverse tasks and dataset sizes, setting a new state of the art in visual prompt tuning. Our code is available at https://github.com/iamjaekyun/vipamin.

Figures (11)

• Figure 1: (a) Nineteen VTAB-1k tasks ordered by a proxy for task similarity to the pretraining dataset, measured by the relative accuracy ratio of linear probing compared to full fine-tuning (LP ratio). Representative images from four datasets---dSprites/loc, smallNORB/azi, Caltech101, and Sun397---are shown to the right. (b) Relative accuracy ratio of VPT compared to full fine-tuning (VPT ratio) across $19$ VTAB datasets, sorted by the LP ratio. (c) Accuracy comparison of VPT between training with the full dataset and few-shot learning (8 images per class), evaluated on the CUB-200-2011 dataset using supervised and self-supervised (MoCo-v3) pretrained weights.
• Figure 2: Failure modes of VPT. (a) Prompt attention entropy of the fully trained model for each VTAB dataset, sorted by its LP ratio. The topmost blue line represents the maximum prompt attention entropy attainable, $\ln(N_e)$. (b) Subspace projection energy of ${\mathbf{P}}_0{\mathbf{W}}_V$ onto $\text{SA}({\mathbf{X}}_0)$, tracked during training on dSprites/loc (the most dissimilar task). Both figures are the training results of MoCo-v3 pretrained model.
• Figure 3: Coherence of embeddings in key space. For an image of a bakery, we show matched tokens for two randomly sampled prompts by highlighting the top-$k$ indices ($k=60$) in red.
• Figure 4: (a) MAE-pretrained ViT-B/L/H are evaluated on Oxford Flowers102 from FGVC. (b) We assess the effect of prompt length in MoCo-v3 on CIFAR-100. (c) We evaluate the performance of VIPAMIN on MAE across VTAB-1k, varying the hyperparameters $k$ and $\lambda$. Datasets are ordered by LP ratio from lowest to highest. We select the learning rate that yields the highest validation accuracy and normalize the resulting top-1 accuracy to the $[0, 1]$ range for each dataset (column-wise). A rough trend of the best-performing hyperparameter combinations is indicated by the red arrow.
• Figure 5: Red regions indicate their critical role in the model's class prediction. The cases of $k=2$ and $k=128$ correspond to VIPAMIN models trained with their respective hyperparameter, $k$.