MSCI: Addressing CLIP's Inherent Limitations for Compositional Zero-Shot Learning

TL;DR

This work targets compositional zero-shot learning (CZSL) by addressing CLIP’s weakness in capturing fine-grained local features. It introduces MSCI, which uses two self-adaptive aggregators to extract local (low-level) and global (high-level) visual information from intermediate CLIP layers and integrates this information into textual prompts through a staged cross-modal interaction mechanism. A dynamic fusion module adjusts the relative influence of local and global cues across different state–object combinations, improving adaptability and performance. Experiments on MIT-States, UT-Zappos, and C-GQA demonstrate state-of-the-art results in both closed-world and open-world CZSL settings, highlighting the practical impact of leveraging intermediate visual features for fine-grained compositional reasoning.

Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen state-object combinations by leveraging known combinations. Existing studies basically rely on the cross-modal alignment capabilities of CLIP but tend to overlook its limitations in capturing fine-grained local features, which arise from its architectural and training paradigm. To address this issue, we propose a Multi-Stage Cross-modal Interaction (MSCI) model that effectively explores and utilizes intermediate-layer information from CLIP's visual encoder. Specifically, we design two self-adaptive aggregators to extract local information from low-level visual features and integrate global information from high-level visual features, respectively. These key information are progressively incorporated into textual representations through a stage-by-stage interaction mechanism, significantly enhancing the model's perception capability for fine-grained local visual information. Additionally, MSCI dynamically adjusts the attention weights between global and local visual information based on different combinations, as well as different elements within the same combination, allowing it to flexibly adapt to diverse scenarios. Experiments on three widely used datasets fully validate the effectiveness and superiority of the proposed model. Data and code are available at https://github.com/ltpwy/MSCI.

Figures (5)

• Figure 1: Core idea of our work. By effectively leveraging the rich local details in the lower-level features of the visual encoder, CLIP's ability to capture fine-grained local information can be enhanced.
• Figure 2: The general framework of the MSCI model. We design two self-adaptive aggregators to extract local and global visual information from low-level and high-level visual features, respectively, and integrate this information into the prompt embeddings through stage-wise cross-modal interactions. Additionally, a fusion module is introduced to regulate the influence of local and global information on the generation of the final prompt embeddings.
• Figure 3: Illustration of the low-level feature aggregator.
• Figure 4: The impact of the number of aggregated layers $N$ on AUC.
• Figure 5: Qualitative analysis of the proposed model.