Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition

TL;DR

This work addresses the challenge of long-tailed multi-label visual recognition by leveraging vision-language models. It introduces CapNet, which adapts CLIP with class-specific soft prompts and a three-layer GCN to model robust label correlations derived from CLIP's textual encoder, combined with a distribution-balanced focal loss, test-time ensembling, and parameter-efficient fine-tuning. The approach achieves state-of-the-art results on VOC-LT, COCO-LT, and NUS-WIDE, especially boosting tail-class recall while preserving head-class performance. The findings demonstrate that language-based priors and graph-guided semantic propagation effectively mitigate imbalanced data, offering a practical and scalable path for real-world LTML applications across diverse backbones and open-set scenarios.

Abstract

Long-tailed multi-label visual recognition poses a significant challenge, as images typically contain multiple labels with highly imbalanced class distributions, leading to biased models that favor head classes while underperforming on tail classes. Recent efforts have leveraged pre-trained vision-language models, such as CLIP, alongside long-tailed learning techniques to exploit rich visual-textual priors for improved performance. However, existing methods often derive semantic inter-class relationships directly from imbalanced datasets, resulting in unreliable correlations for tail classes due to data scarcity. Moreover, CLIP's zero-shot paradigm is optimized for single-label image-text matching, making it suboptimal for multi-label tasks. To address these issues, we propose the correlation adaptation prompt network (CAPNET), a novel end-to-end framework that explicitly models label correlations from CLIP's textual encoder. The framework incorporates a graph convolutional network for label-aware propagation and learnable soft prompts for refined embeddings. It utilizes a distribution-balanced Focal loss with class-aware re-weighting for optimized training under imbalance. Moreover, it improves generalization through test-time ensembling and realigns visual-textual modalities using parameter-efficient fine-tuning to avert overfitting on tail classes without compromising head class performance. Extensive experiments and ablation studies on benchmarks including VOC-LT, COCO-LT, and NUS-WIDE demonstrate that CAPNET achieves substantial improvements over state-of-the-art methods, validating its effectiveness for real-world long-tailed multi-label visual recognition.

Figures (8)

• Figure 1: Comparison between zero-shot CLIP radford2021learning and our proposed CapNet. Image embeddings are represented by red four-angle stars, and text embeddings by circles. The proximity indicates the higher probability that the category of text embeddings matches the image.
• Figure 2: Overall framework of our CapNet method for long-tailed multi-label visual recognition. The image encoder is fine-tuned using either full fine-tuning or parameter-efficient fine-tuning.
• Figure 3: Comparison of different correlation matrices on VOC-LT and COCO-LT datasets.
• Figure 4: Performance (mAP%) comparisons with different hyperparameter $s$ on COCO-LT and VOC-LT datasets.
• Figure 5: Performance (total mAP%) comparisons comparisons with different values of $\tau^{\prime}$ on COCO-LT and VOC-LT datasets.