PVLR: Prompt-driven Visual-Linguistic Representation Learning for Multi-Label Image Recognition

TL;DR

PVLR tackles multi-label image recognition by fully exploiting linguistic knowledge through dual prompting and bidirectional visual-linguistic interactions. The Knowledge-Aware Prompting (KAP) and Context-Aware Prompting (CAP) generate both general and task-specific label semantics, which are deeply fused via the Interaction and Fusion Module (IFM) and further refined by Dual-Modal Attention (DMA) to create adaptive, context-aware category centers. The approach yields state-of-the-art performance across MS-COCO, PASCAL VOC 2007, and NUS-WIDE, with ablations confirming the contributions of each module and the robustness of input-adaptive label representations. This framework demonstrates the value of literature-informed language models in practical, real-world multi-label recognition tasks and offers a blueprint for integrating strong linguistic priors with visual representations.

Abstract

Multi-label image recognition is a fundamental task in computer vision. Recently, vision-language models have made notable advancements in this area. However, previous methods often failed to effectively leverage the rich knowledge within language models and instead incorporated label semantics into visual features in a unidirectional manner. In this paper, we propose a Prompt-driven Visual-Linguistic Representation Learning (PVLR) framework to better leverage the capabilities of the linguistic modality. In PVLR, we first introduce a dual-prompting strategy comprising Knowledge-Aware Prompting (KAP) and Context-Aware Prompting (CAP). KAP utilizes fixed prompts to capture the intrinsic semantic knowledge and relationships across all labels, while CAP employs learnable prompts to capture context-aware label semantics and relationships. Later, we propose an Interaction and Fusion Module (IFM) to interact and fuse the representations obtained from KAP and CAP. In contrast to the unidirectional fusion in previous works, we introduce a Dual-Modal Attention (DMA) that enables bidirectional interaction between textual and visual features, yielding context-aware label representations and semantic-related visual representations, which are subsequently used to calculate similarities and generate final predictions for all labels. Extensive experiments on three popular datasets including MS-COCO, Pascal VOC 2007, and NUS-WIDE demonstrate the superiority of PVLR.

Figures (8)

• Figure 1: Paradigm comparison. (a) Previous methods chen2019learningyou2020crosswang2020multizhu2022twozhu2023scene simply adopt category names to extract text features and they apply a one-way interaction to yield label representations. Then $C$ classifiers are trained to perform recognition. (b) Our method treats the two modalities of equal importance, where we propose dual prompting to extract text features and perform dual interaction to generate context-aware label representations and semantic-related visual representations.
• Figure 2: Overview of the proposed PVLR framework.1) KAP adopts fixed prompts to facilitate the extraction of semantic knowledge from the language model. 2) CAP utilizes learnable prompts to integrate context clues into label representations. 3) IFM deeply aggregates the information learned by KAP and CAP through channel interaction. And $\mathcal{L}_{KCR}$ is measured to enhance the generalization. 4) DMA performs bidirectional attention to generate context-aware label representations and semantic-related visual representations, respectively. The context-aware label representations are then regarded as the classification weights and the prediction is based on the similarity between these two representations, which achieves input-adaptive category centers.
• Figure 3: Visualization of "Classifier Learning" approach and our proposed PVLR. We present several labels for demonstration and those scores highlighted in red are not successfully recognized. PVLR can accurately perceive and localize small objects in a) and b), similar objects in c), and target objects in even complex context in d).
• Figure 4: Ablation study (%) on the IFM. Results are reported on MS-COCO (left) and NUS-WIDE (right).
• Figure 5: Sensitivity analysis (%) of $\lambda$. Results are reported on MS-COCO (left) and NUS-WIDE (right).