Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model

TL;DR

Vision-language foundation models like CLIP show strong zero-shot capabilities but suffer from decision shortcuts in fine-grained tasks. The authors introduce Spurious Feature Eraser (SEraser), a test-time prompt-tuning framework that erases spurious features by optimizing prompts over auxiliary images, pushing predictions toward a uniform distribution on spurious cues. They formalize a CLIP-style zero-shot setting and a KL-based loss $\mathcal{L}_e = \mathrm{KL}(\hat{P}_e(y|x_e) \| q)$ to discourage reliance on non-causal features, and they propose the S2E evaluation paradigm to robustly assess shortcut resilience. Across real and simulated datasets and multiple backbones, SEraser consistently improves zero-shot accuracy, especially worst-group performance, without modifying model weights, and a flexible auxiliary-image construction strategy enhances practicality in diverse tasks.

Abstract

Vision-language foundation models have exhibited remarkable success across a multitude of downstream tasks due to their scalability on extensive image-text paired data. However, these models also display significant limitations when applied to downstream tasks, such as fine-grained image classification, as a result of ``decision shortcuts'' that hinder their generalization capabilities. In this work, we find that the CLIP model possesses a rich set of features, encompassing both \textit{desired invariant causal features} and \textit{undesired decision shortcuts}. Moreover, the underperformance of CLIP on downstream tasks originates from its inability to effectively utilize pre-trained features in accordance with specific task requirements. To address this challenge, we propose a simple yet effective method, Spurious Feature Eraser (SEraser), to alleviate the decision shortcuts by erasing the spurious features. Specifically, we introduce a test-time prompt tuning paradigm that optimizes a learnable prompt, thereby compelling the model to exploit invariant features while disregarding decision shortcuts during the inference phase. The proposed method effectively alleviates excessive dependence on potentially misleading spurious information. We conduct comparative analysis of the proposed method against various approaches which validates the significant superiority.

Figures (4)

• Figure 1: Observation: The attentive regions obtained by Grad-CAM. By eliminating beach background, CLIP changes its answer to be correct. This indicates the CLIP has learned a rich set of features (CLIP knows the object should be a spider), but the spurious features lead to decision shortcuts.
• Figure 2: Framework of Spurious Feature Eraser (SEraser). Given a test sample, we first generate auxiliary images according to different strategies for test-time prompt tuning (termed as Eraser), then conduct zero-shot classification on the test sample with the optimized prompt from Eraser.
• Figure 3: Pipeline of S2E and the predictions over classes of CLIP on different images. We can find that CLIP can correctly classify the objects on foreground, but its predictions on the images with background tend to be wrong when there are serious decision shortcuts on the background.
• Figure 4: Performance under different patch sampling strategies.