Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin

TL;DR

This work tackles pseudolabel imbalance in vision-language model adaptation by identifying concept mismatch and concept confusion as key causes. It introduces CAP, a three-part framework combining concept alignment (with iterative clustering and LLM-enhanced descriptions) and a confusion-aware calibrated margin (M = S ⊙ m with a margin loss Lm) to promote discriminative, balanced predictions, plus independent adapters for robust fine-tuning on unlabeled data. The method demonstrates strong, data-efficient gains across six benchmarks and three learning paradigms, achieving a relative improvement of 6.29% over the SoTA method. These results suggest CAP effectively mitigates semantic gaps in zero-shot pseudolabeling, enabling more reliable downstream adaptation of VLMs using unlabeled data.

Abstract

Adapting vision-language models (VLMs) to downstream tasks with pseudolabels has gained increasing attention. A major obstacle is that the pseudolabels generated by VLMs tend to be imbalanced, leading to inferior performance. While existing methods have explored various strategies to address this, the underlying causes of imbalance remain insufficiently investigated. To fill this gap, we delve into imbalanced pseudolabels and identify two primary contributing factors: concept mismatch and concept confusion. To mitigate these two issues, we propose a novel framework incorporating concept alignment and confusion-aware calibrated margin mechanisms. The core of our approach lies in enhancing underperforming classes and promoting balanced predictions across categories, thus mitigating imbalance. Extensive experiments on six benchmark datasets with three learning paradigms demonstrate that the proposed method effectively enhances the accuracy and balance of pseudolabels, achieving a relative improvement of 6.29% over the SoTA method. Our code is avaliable at https://anonymous.4open.science/r/CAP-C642/

Figures (15)

• Figure 1: left: The lowest 5 per-class accuracies in RESISC45, right: The distribution of samples of them in clusters. The pink bar represents samples in the cluster in which they appear most frequently, the gray bar represents samples appear in other clusters.
• Figure 2: left: concept mismatch. right: concept confusion. Please see Appendix \ref{['apd-a']} for realistic examples of them.
• Figure 3: The process of concept alignment. We first take an iterative clustering strategy to detect the concept-mismatched classes. We then utilize LLMs to generate enhanced descriptions for them, and obtain images with top-$k$ similar image features to the enhanced text feature each class as pseudolabeled samples.
• Figure 4: Density curve of confidence score for samples in concept-confused groups by left: zero-shot CLIP and right: CLIP fine-tuned with confusion-aware calibrated margin.
• Figure 5: Overview of our framework. In the initialization stage, we use concept alignment to obtain $\mathcal{D}_\mathrm{PL}$. In the fine-tuning stage, we deploy the main adapter and pseudo adapter to the visual branch, allowing for separate learning from pseudolabeled and unlabeled samples, and we utilize the confusion-aware calibrated margin matrix $\mathbf{M}$ to compute the loss.