Sample-specific Masks for Visual Reprogramming-based Prompting

TL;DR

This work tackles the generalization limits of visual reprogramming caused by a shared input mask by introducing Sample-specific Multi-channel Masks (SMM). SMM combines a lightweight mask generator with patch-wise interpolation to produce per-sample three-channel masks, enabling tailored input perturbations while keeping the pretrained model fixed. The authors prove, within a PAC framework, that SMM has a smaller approximation error than shared-mask VR and demonstrate substantial empirical gains across ResNet and ViT on 11 datasets, including improvements over strong baselines and compatibility with LoRA finetuning. The approach offers a scalable, parameter-efficient means to repurpose pretrained models for diverse target tasks, broadening the practical impact of VR.

Abstract

Visual reprogramming (VR) is a prompting technique that aims to re-purpose a pre-trained model (e.g., a classifier on ImageNet) to target tasks (e.g., medical data prediction) by learning a small-scale pattern added into input images instead of tuning considerable parameters within the model. The location of the pattern within input samples is usually determined by a pre-defined mask shared across all samples. In this paper, we show that the shared mask potentially limits VR's generalization and increases its approximation error due to the lack of sample-level adaptation. Motivated by this finding, we design a new framework for VR called sample-specific multi-channel masks (SMM). Specifically, SMM employs a lightweight ConvNet and patch-wise interpolation to generate sample-specific three-channel masks instead of a shared and pre-defined mask. Since we generate different masks for individual samples, SMM is theoretically shown to reduce approximation error for the target tasks compared with existing state-of-the-art VR methods. We also empirically demonstrate its performance gain on both ResNet and ViT. The success of SMM further highlights the broader applicability of VR in leveraging the latent knowledge of pre-trained models for various target tasks. Our code is available at https://github.com/tmlr-group/SMM.

Figures (23)

• Figure 1: Drawback of shared masks over individual images. We demonstrate the use of watermarking wang2022watermarking, a representative VR method, to re-purpose an ImageNet-pretrained classifier for the OxfordPets dataset, with different shared masks (full, medium, and narrow) in VR. An evaluation of classification confidence across three cat images — Sphynx, Abyssinian, and Bengal — indicates a sample-specific mask preference: Sphynx with medium, Abyssinian with full, and Bengal with narrow. It shows that different masks are needed for individual images.
• Figure 2: Drawback of shared masks in the statistical view. Optimal learning methods like finetuning usually result in loss decreases for all samples (see the blue part). But when applying the same mask in reprogramming, part of the loss changes are observed to be positive (see the red part) according to the distribution of [final loss - initial loss], which means the training loss for some samples even rises.
• Figure 3: Comparison between (a) existing methods and (b) our method. Previous padding-based reprogramming adds zeros around the target image, while resizing-based reprogramming adjusts image dimensions to fit the required input size. Both methods use a pre-determined shared mask to indicate the valid location of pattern $\delta$. Our method, on the other hand, takes a more dynamic and tailored approach. We resize each target image and apply a different three-channel mask accordingly, driven by a lightweight $f_{\rm mask}$ with an interpolation up-scaling module, allowing for more variability in individual samples.
• Figure 4: Comparative results of different patch sizes ($2^l$). ResNet-18 is used as the pre-trained model as an example.
• Figure 5: Visual results of trained VR on the Flowers102 dataset. To show the difference in results, the original image, result image and SMM adopt histogram equalization. ResNet-18 is used as the pre-trained model as an example. Other visualization results and further analysis are included in Appendix \ref{['visualresults']}.

Theorems & Definitions (6)

• Definition 4.1: Approximation Error
• Theorem 4.2
• Proposition 4.3
• proof
• Proposition 2.1
• proof