Prompting Language-Informed Distribution for Compositional Zero-Shot Learning

TL;DR

This work tackles CZSL by combining language-informed distributions, generated from LLMs, with CLIP-based visual-language alignment to produce diverse and informative class representations. The PLID framework augments text embeddings with LLM-derived descriptions and Gaussian-distributed distribution supports, andively couples this with a visual-language primitive decomposition (VLPD) that decomposes visual features into state and object pathways. A stochastic logit mixup fuses directly predicted compositions with recomposed, primitive-based predictions, enabling robust zero-shot generalization. Across MIT-States, UT-Zappos, and C-GQA, PLID achieves state-of-the-art results in both closed-world and open-world settings, with ablations confirming the value of language-informed distributions, VLPD, and stochastic fusion for CZSL performance and generalization.

Abstract

Compositional zero-shot learning (CZSL) task aims to recognize unseen compositional visual concepts, e.g., sliced tomatoes, where the model is learned only from the seen compositions, e.g., sliced potatoes and red tomatoes. Thanks to the prompt tuning on large pre-trained visual language models such as CLIP, recent literature shows impressively better CZSL performance than traditional vision-based methods. However, the key aspects that impact the generalization to unseen compositions, including the diversity and informativeness of class context, and the entanglement between visual primitives, i.e., state and object, are not properly addressed in existing CLIP-based CZSL literature. In this paper, we propose a model by prompting the language-informed distribution, aka., PLID, for the CZSL task. Specifically, the PLID leverages pre-trained large language models (LLM) to (i) formulate the language-informed class distributions which are diverse and informative, and (ii) enhance the compositionality of the class embedding. Moreover, a visual-language primitive decomposition (VLPD) module is proposed to dynamically fuse the classification decisions from the compositional and the primitive space. Orthogonal to the existing literature of soft, hard, or distributional prompts, our method advocates prompting the LLM-supported class distributions, leading to a better zero-shot generalization. Experimental results on MIT-States, UT-Zappos, and C-GQA datasets show the superior performance of the PLID to the prior arts. Our code and models are released: https://github.com/Cogito2012/PLID.

Figures (8)

• Figure 1: Challenges of compositional recognition.(a) images of the same compositional class appear differently due to diverse visual backgrounds or foregrounds. (b)red tomatoes and sliced tomatoes are visually correlated because 1) both are tomatoes object, and 2) the object tomatoes is inherently entangled with the state red, resulting in the need of primitive decomposition.
• Figure 2: Overview of $\mathbb{PLID}$. The model is developed for the CZSL task by aligning the semantics of image $\mathbf{x}$ (e.g., image on the right) and compositional class $y=(s,o)$ (e.g., "red apple") via a frozen CLIP CLIP. It constructs language-informed text distributions in both compositional and primitive (attribute and object) spaces (middle part) by soft prompting and LLM-generated class descriptions (left part). The features of the image and text are enhanced by text and visual feature enhancement (TFE and VFE). Eventually, the compositional decisions from the two spaces are fused as the prediction.
• Figure 3: Prompting for intra- and inter-class covariance optimization.
• Figure 4: VLPD for recomposing.
• Figure 5: Impact of $M$ and $N$. We set $N=8$ for the Fig. \ref{['fig:auc_m']}, while we set $M=64$ for the Fig. \ref{['fig:auc_n']}.