Prompt-based Adaptation in Large-scale Vision Models: A Survey

TL;DR

This survey clarifies the landscape of prompt-based adaptation (PA) for large-scale vision models by distinguishing Visual Prompting (VP) from Visual Prompt Tuning (VPT) and presenting a unified taxonomy that separates where prompts act (input versus internal) from how prompts are obtained (learnable, generative, or non-learnable) and by injection granularity (pixel-level vs token-level). It surveys a broad range of foundational tasks (segmentation, restoration, compression, multi-modal tasks) and diverse domain applications (medical imaging, remote sensing, robotics, industrial inspection, autonomous driving, 3D LiDAR, video, underwater environments), highlighting representative methods, efficiency considerations, and real-world constraints such as data scarcity, distribution shifts, and API access. The paper also discusses practical constraints (data, dynamic environments, resource/access limits), domain-specific deployments, and critical trustworthy-AI issues (robustness, fairness, privacy, and security), and it offers foundational analysis and theory to motivate further development. By delivering a structured, deployment-oriented roadmap, the survey aims to guide researchers and practitioners in selecting PA paradigms and designing robust, efficient prompts for vision backbones across diverse settings.

Abstract

In computer vision, Visual Prompting (VP) and Visual Prompt Tuning (VPT) have recently emerged as lightweight and effective alternatives to full fine-tuning for adapting large-scale vision models within the ``pretrain-then-finetune'' paradigm. However, despite rapid progress, their conceptual boundaries remain blurred, as VP and VPT are frequently used interchangeably in current research, reflecting a lack of systematic distinction between these techniques and their respective applications. In this survey, we revisit the designs of VP and VPT from first principles, and conceptualize them within a unified framework termed Prompt-based Adaptation (PA). We provide a taxonomy that categorizes existing methods into learnable, generative, and non-learnable prompts, and further organizes them by injection granularity -- pixel-level and token-level. Beyond the core methodologies, we examine PA's integrations across diverse domains, including medical imaging, 3D point clouds, and vision-language tasks, as well as its role in test-time adaptation and trustworthy AI. We also summarize current benchmarks and identify key challenges and future directions. To the best of our knowledge, we are the first comprehensive survey dedicated to PA's methodologies and applications in light of their distinct characteristics. Our survey aims to provide a clear roadmap for researchers and practitioners in all area to understand and explore the evolving landscape of PA-related research.

Figures (4)

• Figure 1: A taxonomy of Prompt-based Adaptation (PA) in Large Vision Models.
• Figure 2: Comparison of transfer learning and prompt-based adaptation methods. (a) Current transfer learning protocols are grouped by tuning scope: Full fine-tuning, head-oriented, and backbone-oriented approaches. (b) Visual Prompt Tuning (VPT) freezes the backbone and optimizes additional prompt tokens together with the head. (c) Visual Prompting (VP) instead modifies the input space by adding prompts (which can be fixed or learnable), while keeping the backbone frozen and training a lightweight head.
• Figure 3: Illustration of different variants of Visual Prompt Tuning (VPT, see §\ref{['sec:VPT_new']}). Left: VPT-Learnable (Shallow), where prompt tokens are only added at the first layer. Middle: VPT-Learnable (Deep), where prompt tokens are injected at every transformer layer. Right: VPT-Generated, where a generator produces instance-adaptive prompt tokens that are then inserted.
• Figure 4: Illustration of Visual Prompting (VP) variants (see §\ref{['subsec:VP']}). (Left) VP-Fixed: prompts are predefined (e.g., boxes, points, text hints) and directly attached to the input without training. (Middle) VP-Learned: learnable prompts in the pixel space are optimized jointly with the task head while the backbone remains frozen. (Right) VP-Generated: a generator network produces instance-adaptive prompts for each input image, offering higher flexibility.