Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations

TL;DR

This work systematically study the existence of spurious correlation on CLIP and CILP+ERM, and advocates a lightweight representation calibration method for fine-tuning CLIP, by first generating a calibration set using the pretrained CLIP, and then calibrating representations of samples within this set through contrastive learning, all without the need for group labels.

Abstract

Fine-tuning pre-trained vision-language models, like CLIP, has yielded success on diverse downstream tasks. However, several pain points persist for this paradigm: (i) directly tuning entire pre-trained models becomes both time-intensive and computationally costly. Additionally, these tuned models tend to become highly specialized, limiting their practicality for real-world deployment; (ii) recent studies indicate that pre-trained vision-language classifiers may overly depend on spurious features -- patterns that correlate with the target in training data, but are not related to the true labeling function; and (iii) existing studies on mitigating the reliance on spurious features, largely based on the assumption that we can identify such features, does not provide definitive assurance for real-world applications. As a piloting study, this work focuses on exploring mitigating the reliance on spurious features for CLIP without using any group annotation. To this end, we systematically study the existence of spurious correlation on CLIP and CLIP+ERM. We first, following recent work on Deep Feature Reweighting (DFR), verify that last-layer retraining can greatly improve group robustness on pretrained CLIP. In view of them, we advocate a lightweight representation calibration method for fine-tuning CLIP, by first generating a calibration set using the pretrained CLIP, and then calibrating representations of samples within this set through contrastive learning, all without the need for group labels. Extensive experiments and in-depth visualizations on several benchmarks validate the effectiveness of our proposals, largely reducing reliance and significantly boosting the model generalization.

Figures (10)

• Figure 1: $t$-SNE and UMAP visualizations for pre-trained CLIP, ERM-tuned CLIP, and CFR (ours) on Waterbirds. We observe that both the pre-trained and ERM-tuned CLIP exhibit noticeable spurious correlations, with feature separations inappropriately aligned with spurious attributes, specifically the background, rather than the target class. In contrast, our method, as visualized through $t$-SNE and UMAP, demonstrates a significantly improved class separations, underscoring the robustness of our method in reducing spurious correlations.
• Figure 2: Layer-by-layer GradCAM analysis of the CLIP-ResNet50. Each row starts with the original image on the left, followed by four GradCAM visualizations corresponding to the four successive layers of the ResNet-50, with the depth of the layers increasing from left to right.
• Figure 3: Comparison of methods using the CLIP-ResNet50 architecture on four benchmark datasets. We use Worst Group Accuracy to evaluate the performance for various methods, including ERM, semi-supervised baselines (i.e., AFR qiu2023simple, CnC zhang2022correct, JTT liu2021just), and our proposed methods. We observe that CFR combined with the sample selection strategies (i.e., {DPS, RPS}$\times${RNS,NNS}) outperforms all semi-supervised baselines across all benchmarks.
• Figure 4: Training-validation curves of various semi-supervised methods using CLIP-ResNet. We plot WGA on a validation dataset at regular intervals of 25 epochs throughout the training process. Results are averaged across 3 random seeds.
• Figure 5: The illustration of our proposed method CFR.CFR decompose a lightweight representation calibration into two steps. (1) Calibration Set Formation: When a pre-trained CLIP is fine-tuned using ERM, this ERM-tuned CLIP with the frozen weights constructs a calibration set from the training data, as introduced in Sec. \ref{['sec:method calibration set formulation']} (Main Context). This set comprises pivotal anchor points, with each sample selected based on its misclassification by the ERM-tuned CLIP. These anchors play a crucial role in refining the robustness across the dataset. (2) Contrastive Feature Recalibration: Utilizing the curated calibration set, CFR focuses on refining sample representations. This process involves aligning them more closely with the centroid of their respective class in the feature space while simultaneously distancing them from centroids of opposing classes. Such a recalibration is efficiently performed via a contrastive loss. Details about the positive and negative sample selection strategies used in CFR are discussed in Sec. \ref{['sec:method feature recalibration']} (Main Context).