Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attribute-based Visual Reprogramming for Vision-Language Models

Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, Feng Liu

TL;DR

AttrVR tackles the limitations of template-prompt supervision in VR for CLIP by introducing DesAttrs and DistAttrs as attribute-based supervision, generated via LLMs. It employs a sample-specific $k$-nearest attribute strategy to continuously refine VR patterns across epochs, guided by a weighted attribute embedding similarity and gradient-based optimization. The authors prove that DesAttrs reduce intra-class variance while DistAttrs boost inter-class separation, and validate these gains with extensive experiments across 12 tasks and multiple backbones, showing consistent improvements over prior VR methods. The approach demonstrates that leveraging descriptive and distinctive attributes can more effectively align visual and textual representations in vision-language models, enabling stronger transfer to diverse downstream domains. Open-source code is provided for reproducibility and broader adoption.

Abstract

Visual reprogramming (VR) reuses pre-trained vision models for downstream image classification tasks by adding trainable noise patterns to inputs. When applied to vision-language models (e.g., CLIP), existing VR approaches follow the same pipeline used in vision models (e.g., ResNet, ViT), where ground-truth class labels are inserted into fixed text templates to guide the optimization of VR patterns. This label-based approach, however, overlooks the rich information and diverse attribute-guided textual representations that CLIP can exploit, which may lead to the misclassification of samples. In this paper, we propose Attribute-based Visual Reprogramming (AttrVR) for CLIP, utilizing descriptive attributes (DesAttrs) and distinctive attributes (DistAttrs), which respectively represent common and unique feature descriptions for different classes. Besides, as images of the same class may reflect different attributes after VR, AttrVR iteratively refines patterns using the $k$-nearest DesAttrs and DistAttrs for each image sample, enabling more dynamic and sample-specific optimization. Theoretically, AttrVR is shown to reduce intra-class variance and increase inter-class separation. Empirically, it achieves superior performance in 12 downstream tasks for both ViT-based and ResNet-based CLIP. The success of AttrVR facilitates more effective integration of VR from unimodal vision models into vision-language models. Our code is available at https://github.com/tmlr-group/AttrVR.

Attribute-based Visual Reprogramming for Vision-Language Models

TL;DR

AttrVR tackles the limitations of template-prompt supervision in VR for CLIP by introducing DesAttrs and DistAttrs as attribute-based supervision, generated via LLMs. It employs a sample-specific -nearest attribute strategy to continuously refine VR patterns across epochs, guided by a weighted attribute embedding similarity and gradient-based optimization. The authors prove that DesAttrs reduce intra-class variance while DistAttrs boost inter-class separation, and validate these gains with extensive experiments across 12 tasks and multiple backbones, showing consistent improvements over prior VR methods. The approach demonstrates that leveraging descriptive and distinctive attributes can more effectively align visual and textual representations in vision-language models, enabling stronger transfer to diverse downstream domains. Open-source code is provided for reproducibility and broader adoption.

Abstract

Visual reprogramming (VR) reuses pre-trained vision models for downstream image classification tasks by adding trainable noise patterns to inputs. When applied to vision-language models (e.g., CLIP), existing VR approaches follow the same pipeline used in vision models (e.g., ResNet, ViT), where ground-truth class labels are inserted into fixed text templates to guide the optimization of VR patterns. This label-based approach, however, overlooks the rich information and diverse attribute-guided textual representations that CLIP can exploit, which may lead to the misclassification of samples. In this paper, we propose Attribute-based Visual Reprogramming (AttrVR) for CLIP, utilizing descriptive attributes (DesAttrs) and distinctive attributes (DistAttrs), which respectively represent common and unique feature descriptions for different classes. Besides, as images of the same class may reflect different attributes after VR, AttrVR iteratively refines patterns using the -nearest DesAttrs and DistAttrs for each image sample, enabling more dynamic and sample-specific optimization. Theoretically, AttrVR is shown to reduce intra-class variance and increase inter-class separation. Empirically, it achieves superior performance in 12 downstream tasks for both ViT-based and ResNet-based CLIP. The success of AttrVR facilitates more effective integration of VR from unimodal vision models into vision-language models. Our code is available at https://github.com/tmlr-group/AttrVR.
Paper Structure (27 sections, 5 theorems, 38 equations, 11 figures, 14 tables, 1 algorithm)

This paper contains 27 sections, 5 theorems, 38 equations, 11 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Attribute-based Visual Reprogramming
  5. Understanding the Effects of Attributes
  6. Experiments
  7. Conclusion
  8. Appendix 1: More Training Information
  9. The Problem Setting of VR for CLIP
  10. Implement Details
  11. Generating DesAttrs and DistAttrs
  12. Dataset Details
  13. Training VR Patterns
  14. Details About Zero-shot AttrZS
  15. Appendix 2: More Theoretical Justification
  16. ...and 12 more sections

Key Result

Lemma 1

Let ${\mathcal{A}}_{\rm des}(y) \subseteq {\mathcal{A}}(y)$ be the set of descriptive attributes for class $y$ as with Definition def:Des. Let $\Sigma_{\rm A}$ and $\Sigma_{\rm L}$ be the covariance matrices of the embeddings optimized with respect to ${\mathcal{A}}_{\rm des}(y)$ and $y$, respective

Figures (11)

  • Figure 1: T-SNE visualization results of (a) embeddings of images with label-based (i.e., 'This is a photo of [label]') VR and (b) embeddings of text DesAttrs and DistAttrs for classes 'British Shorthair' and 'Russia Blue'. Examples of images with VR or attributes are shown below. Misclassifications occur in images with label-based VR, whereas attributes are easily distinguishable.
  • Figure 2: The comparison of (a) previous label-based VR and (b) our attribute-based VR. Previous VR methods use fixed template-prompted ground-truth labels for all samples to optimize the VR pattern $\delta$ (using Eq. (\ref{['eq:clip_prob']}) and Eq. (\ref{['eq:inputvr']})), whereas our method iteratively selects $k$ nearest DesAttrs and DistAttrs for individual samples in each epoch to optimize the VR pattern $\delta$ (using Eq. (\ref{['eq:weighted_clip']})).
  • Figure 3: Accuracy comparison of different VR methods trained on different shots from [1, 4, 8, 16, 32]. Pre-trained ViT-B16-based CLIP is used. The striped area indicates the error bars.
  • Figure 4: Visualization of images with AttrVR patterns, and their nearest DesAttrs and DistAttrs, using the ViT-B16-based CLIP as the pre-trained model. Two images labeled 'Globe Thistle' from 'Flowers' and two labeled 'Banded' from 'Texture' are chosen as examples (more in Appendix \ref{['app:vis']}).
  • Figure 5: T-SNE visualization results of image embeddings from seven classes in the Flowers task, utilizing the ViT-B16-based CLIP as the pre-trained model. In the first plot, embeddings of zero-shot images are indicated with ZS. The following three plots display embeddings of images with VR patterns, categorized by different training methods and marked as VP, AR, and AttrVR, respectively.
  • ...and 6 more figures

Theorems & Definitions (11)

  • Definition 1: Attributes
  • Definition 2: Descriptive Attributes
  • Definition 3: Distinctive Attributes
  • Definition 4: Class Separability
  • Lemma 1
  • Lemma 2
  • Corollary 1
  • Lemma 3: cf. Lemma \ref{['lem:Des']}
  • proof
  • Lemma 4: cf. Lemma \ref{['lem:Dist']}
  • ...and 1 more