Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition

TL;DR

This work reveals that vision-language models trained with attribute-based cues rely disproportionately on spuriously correlated attributes that co-occur with target categories, harming out-of-distribution generalization. It introduces Spurious Attribute Probing (SAP) to identify and filter out these spurious attributes, and Spurious Attribute Shielding (SAS) to mitigate their influence by adding a subsidiary task with pseudo categories, making SAS a plug-and-play module for various PEFT methods. Empirical results across 11 datasets and multiple generalization tasks show SAP substantially improves generalization of existing attribute-based methods, while SAS achieves state-of-the-art robustness and can complement a wide range of baselines and modalities (images and video). Together, SAP and SAS offer a scalable, training-friendly approach to reduce bias from spurious cues, enhancing robustness without sacrificing in-distribution performance, with practical implications for safer and more reliable vision-language systems.

Abstract

Few-shot adaptation for Vision-Language Models (VLMs) presents a dilemma: balancing in-distribution accuracy with out-of-distribution generalization. Recent research has utilized low-level concepts such as visual attributes to enhance generalization. However, this study reveals that VLMs overly rely on a small subset of attributes on decision-making, which co-occur with the category but are not inherently part of it, termed spuriously correlated attributes. This biased nature of VLMs results in poor generalization. To address this, 1) we first propose Spurious Attribute Probing (SAP), identifying and filtering out these problematic attributes to significantly enhance the generalization of existing attribute-based methods; 2) We introduce Spurious Attribute Shielding (SAS), a plug-and-play module that mitigates the influence of these attributes on prediction, seamlessly integrating into various Parameter-Efficient Fine-Tuning (PEFT) methods. In experiments, SAP and SAS significantly enhance accuracy on distribution shifts across 11 datasets and 3 generalization tasks without compromising downstream performance, establishing a new state-of-the-art benchmark.

Figures (15)

• Figure 1: The phenomenon of Black Sheep in the Herd. We rank attribute weights on VLM predictions using CBMs, with yellow and purple bars to denote spurious and core attributes respectively. In (b), we observe that for vanilla VLMs, 2 out of the top-3 are spurious attributes, heavily influencing decisions. In (c), SAS mitigates this by suppressing the influence of spurious attributes.
• Figure 2: The overview of SAS. In (a), we generate and identify spurious attributes with SAP. In (b), we construct pseudo categories by synthetic data (SD) or retrieval (LAION). In (c), apart from the main objective (i), e.g., cross-entropy loss, we introduce an auxiliary subsidiary task (ii) for learning robust features.
• Figure 3: The average results of three generalization tasks over 11 datasets. The x-axis and y-axis represent in-distribution/base accuracy and out-of-distribution/new accuracy, respectively. We present the out-of-distribution accuracy of vanilla CLIP as a horizontal bar to represent the zero-shot capability. The detailed numerical results are provided in Supp. Mat. \ref{['subsec:E4']}.
• Figure 4: Example samples from test set and counter group. The samples from counter group do not contain spurious attributes, e.g., ${\rm ice}$ or ${\rm sky}$.
• Figure 5: The results for standard few-shot classification on test set and counter group, respectively. Essentially, counter group is a subset of test set where spurious attributes are removed.