AutoVP: An Automated Visual Prompting Framework and Benchmark

TL;DR

This work proposes AutoVP, an end-to-end expandable framework for automating VP design choices, along with 12 downstream image-classification tasks that can serve as a holistic VP-performance benchmark and shows that AutoVP outperforms the best-known current VP methods by a substantial margin.

Abstract

Visual prompting (VP) is an emerging parameter-efficient fine-tuning approach to adapting pre-trained vision models to solve various downstream image-classification tasks. However, there has hitherto been little systematic study of the design space of VP and no clear benchmark for evaluating its performance. To bridge this gap, we propose AutoVP, an end-to-end expandable framework for automating VP design choices, along with 12 downstream image-classification tasks that can serve as a holistic VP-performance benchmark. Our design space covers 1) the joint optimization of the prompts; 2) the selection of pre-trained models, including image classifiers and text-image encoders; and 3) model output mapping strategies, including nonparametric and trainable label mapping. Our extensive experimental results show that AutoVP outperforms the best-known current VP methods by a substantial margin, having up to 6.7% improvement in accuracy; and attains a maximum performance increase of 27.5% compared to linear-probing (LP) baseline. AutoVP thus makes a two-fold contribution: serving both as an efficient tool for hyperparameter tuning on VP design choices, and as a comprehensive benchmark that can reasonably be expected to accelerate VP's development. The source code is available at https://github.com/IBM/AutoVP.

Figures (18)

• Figure 1: Overview and key highlights of AutoVP. The main components of AutoVP are: Input Scaling, which offers three initial input scale options: $\times 0.5$, $\times 1.0$, and $\times 1.5$; Visual Prompt, which pads the prompts to the scaled input image; Pre-trained Classifier, allowing users (or AutoVP) to select from four pre-trained models: ResNet18 he2016deep, ResNeXt101-IG mahajan2018exploring, Swin-T liu2021swin, and CLIP radford2021learning; and Output Label Mapping, offering four label mapping options: Iterative Mapping (IterMap), Frequency Mapping (FreqMap), Semantic Mapping (SemanticMap), and Fully Connected Layer Mapping (FullyMap). Bottom panel: Given a fixed ImageNet-pre-trained classifier (ResNet18), AutoVP outperforms the state-of-the-art (ILM-VP in chen2022understanding) on all 12 different downstream image-classification tasks.
• Figure 2: Data Scalability. The chart presents the average accuracy of AutoVP and LP across the 12 datasets with varying data percentages: 100%, 50%, 25%, 10%, and 1%. The green bar represents the accuracy gains achieved by AutoVP compared to Linear Probing (LP).
• Figure 3: The Tuning Result on Flowers102 Dataset. The color bars represent the accuracy tuning for 3 to 5 epochs (with ASHA early-stop optimization). Points illustrate the test accuracy achieved for the best settings given the pre-trained model and mapping method.
• Figure 4: Models Robustness with CLIP. The comparison of accuracy drop on the CIFAR10-C dataset across AutoVP, ILM-VP, CLIP-VP and LP.
• Figure 5: Accuracy Gains with CLIP. The right side of the chart indicates a higher out-of-distribution (OOD) extent, accompanied by larger gain values. Conversely, the left side shows lower gain values.