Debiased Prompt Tuning in Vision-Language Model without Annotations

TL;DR

Spurious correlations hinder the robustness of prompt-tuned Vision-Language Models (VLMs). The authors introduce Debiased Prompt-Tuning (DPT), a lightweight framework that uses CLIP's zero-shot capabilities to infer pseudo-spurious attribute labels via language prompts and then applies a dynamic group-aware reweighting during prompt tuning. Key formulas include the group weight update $\,\hat{w}_g \ opropto \\frac{1}{N_g} \\exp{[\\eta (1-\\overline{p}_{y|g})]}$ and the final loss $L(\\theta) = \\sum_{(x,y,g) \\in D} w_g l(x,y\\|\\theta)$, with EMA updates for $w_g$. DPT achieves state-of-the-art or competitive worst-group accuracy and reduced robustness gaps on Waterbirds, CelebA, and MetaShift, without needing manual spurious-label annotations and with efficient fine-tuning. The approach also transfers across VLMs and provides improvements for unimodal vision models, highlighting practical impact for robust, real-world deployment of prompt-tuned systems.

Abstract

Prompt tuning of Vision-Language Models (VLMs) such as CLIP, has demonstrated the ability to rapidly adapt to various downstream tasks. However, recent studies indicate that tuned VLMs may suffer from the problem of spurious correlations, where the model relies on spurious features (e.g. background and gender) in the data. This may lead to the model having worse robustness in out-of-distribution data. Standard methods for eliminating spurious correlation typically require us to know the spurious attribute labels of each sample, which is hard in the real world. In this work, we explore improving the group robustness of prompt tuning in VLMs without relying on manual annotation of spurious features. We notice the zero - shot image recognition ability of VLMs and use this ability to identify spurious features, thus avoiding the cost of manual annotation. By leveraging pseudo-spurious attribute annotations, we further propose a method to automatically adjust the training weights of different groups. Extensive experiments show that our approach efficiently improves the worst-group accuracy on CelebA, Waterbirds, and MetaShift datasets, achieving the best robustness gap between the worst-group accuracy and the overall accuracy.

Figures (3)

• Figure 1: The framework of Debiased Prompt-Tuning. On the top-left, we first use VLMs and text descriptions to infer spurious attribute labels and get pseudo-group annotations. On the top-right, misclassified groups are automatically re-weighted, and unbiased text features are obtained by prompt tuning. On the bottom, we revisit the prompt tuning method (Context Optimization) of VLMs and get the image and text features.
• Figure 2: We use Grad-CAM selvaraju2017grad to visualize attention for different approaches (Zero-shot CLIP, CoOp, and Ours) using CLIP-ResNet-50 for Waterbirds and CelebA datasets.
• Figure 3: Analysis on the sensitivity of hyper-parameters $\eta$ on CLIP ViT-L/14. We run experiments and report means over three seeds.