CAMS: Towards Compositional Zero-Shot Learning via Gated Cross-Attention and Multi-Space Disentanglement

TL;DR

CAMS tackles compositional zero-shot learning by addressing the blind spots of global CLIP-based representations. It introduces a Gated Cross-Attention module to extract fine-grained semantic features via latent units and then performs multi-space disentanglement to separate attribute, object, and composition semantics. Training aligns these disentangled semantic representations with prompt-based text representations, delivering strong performance in both closed-world and open-world CZSL across MIT-States, UT-Zappos, and C-GQA, with notable gains in HM and AUC. The approach also demonstrates efficiency advantages and robust ablation results, underscoring its potential for real-world compositional reasoning in vision-language tasks.

Abstract

Compositional zero-shot learning (CZSL) aims to learn the concepts of attributes and objects in seen compositions and to recognize their unseen compositions. Most Contrastive Language-Image Pre-training (CLIP)-based CZSL methods focus on disentangling attributes and objects by leveraging the global semantic representation obtained from the image encoder. However, this representation has limited representational capacity and do not allow for complete disentanglement of the two. To this end, we propose CAMS, which aims to extract semantic features from visual features and perform semantic disentanglement in multidimensional spaces, thereby improving generalization over unseen attribute-object compositions. Specifically, CAMS designs a Gated Cross-Attention that captures fine-grained semantic features from the high-level image encoding blocks of CLIP through a set of latent units, while adaptively suppressing background and other irrelevant information. Subsequently, it conducts Multi-Space Disentanglement to achieve disentanglement of attribute and object semantics. Experiments on three popular benchmarks (MIT-States, UT-Zappos, and C-GQA) demonstrate that CAMS achieves state-of-the-art performance in both closed-world and open-world settings. The code is available at https://github.com/ybyangjing/CAMS.

Figures (6)

• Figure 1: Comparison between CAMS and previous approaches. (a) Existing methods perform attribute–object disentanglement in a single-space using the global semantic representation. (b) CAMS introduces latent units to extract more fine-grained semantic features from high-level encoding blocks, and then employs these features to model separate representation in multi-space, thereby achieving more effective attribute–object disentanglement.
• Figure 2: Illustration of CAMS. First, on the visual side $\mathcal{F}_v$, the global semantic representation $f^g$ of the image is is extracted through the image encoder. Simultaneously, in the last $M$ layers of the image encoder, we introduce a set of Latent Units to extract high-level semantic features of the image using GCA. The semantic features obtained by the Latent Units are then fed into MSD to produce attribute $f^a$, object $f^o$, and compositional $f^c$ semantic representations. Second, on the textual side $\phi_{t}$, we extract attribute $t^a$, object $t^o$, and composition $t^c$ prompt representations through a three-branch prompting mechanism using the CLIP text encoder. Finally, the semantic representations and prompt representations are optimized and aligned via cross-entropy. Where $Tr_a$, $Tr_o$, $Tr_c$ denote the Transformer encoders, and $\pi_a$, $\pi_o$, $\pi_c$ are projection layers. $W_z$ and $U_z$ are the parameter matrices and $\sigma$ is sigmoid function.
• Figure 3: Visualization of baseline representations $f^g$, composition representations $f^c$, attribute representations $f^a$, object representations $f^o$, and combined representations learned by CAMS and Troika representations on UT-Zappos.
• Figure 4: Visualization analysis of Gated Mechanism (GM). Init denotes the regions attended to by the latent units in the initial layer. The second and third columns respectively show the information attended to by the model in the final layer.
• Figure 5: Ablation on Layer Selection $M$.