Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models

TL;DR

This work demonstrates that visual embeddings of pre-trained VLMs exhibit a compositional arrangement, and evaluates the effectiveness of this property in the tasks of compositional classification and group robustness, and proposes a framework that approximates image representations with geometry-aware compositional structures in the latent space.

Abstract

Vision-Language Models (VLMs) learn a shared feature space for text and images, enabling the comparison of inputs of different modalities. While prior works demonstrated that VLMs organize natural language representations into regular structures encoding composite meanings, it remains unclear if compositional patterns also emerge in the visual embedding space. In this work, we investigate compositionality in the image domain, where the analysis of compositional properties is challenged by noise and sparsity of visual data. We address these problems and propose a framework, called Geodesically Decomposable Embeddings (GDE), that approximates image representations with geometry-aware compositional structures in the latent space. We demonstrate that visual embeddings of pre-trained VLMs exhibit a compositional arrangement, and evaluate the effectiveness of this property in the tasks of compositional classification and group robustness. GDE achieves stronger performance in compositional classification compared to its counterpart method that assumes linear geometry of the latent space. Notably, it is particularly effective for group robustness, where we achieve higher results than task-specific solutions. Our results indicate that VLMs can automatically develop a human-like form of compositional reasoning in the visual domain, making their underlying processes more interpretable. Code is available at https://github.com/BerasiDavide/vlm_image_compositionality.

Figures (8)

• Figure 1: Compositional structures in visual embedding space. (left) Pre-trained VLM represents visual inputs of composite meanings in regular geometric shapes. The modularity of these structures enables the separation of the primitive components and the composition of unseen combinations. (right) We evaluate the usefulness of these properties in compositional classification, group robustness, and image generation.
• Figure 2: Sketch of our decomposition method. (top-left) Each concept in $\mathcal{Z} \space=\space \{{\color{red} red}, {\color{blue} blue}\}\times\{\Box, \triangle\}$ is represented by $k=5$ embeddings on a manifold. (bottom) These are mapped in the tangent space where optimal primitive directions are computed as vector means and combined by addition. (top-right) The obtained compositions are mapped back to the manifold to obtain a decomposable approximation of the input embeddings.
• Figure 3: (top) 2-D projections of image embeddings representing the $2\times2$ composite labels of the Waterbirds dataset. (bottom) 3-D projections of image embeddings representing $2\times3$ composite labels with images from the UT-Zappos dataset. Denoised pair representations (marked with a black contour) are computed with $k=1, 5, 30$ randomly selected images.
• Figure 4: Data efficiency of GDE on the group robustness benchmark. Subsets of the full support set are sampled keeping the group ratios fixed. The shaded confidence band shows the standard deviation over five experiments.
• Figure 5: Attribute-object pairs generated using decomposed embeddings with StableDiffusion for the UT-Zappos (first row) and MIT-states (second row) datasets. The two leftmost labels are seen pairs, while the two right-most are unseen pairs. We also generate object-object pairs (third row) blending animal species.

Theorems & Definitions (11)

• Definition 1: Geodesically decomposable embeddings
• Lemma 1
• Proposition 1
• Proposition 2
• Lemma 2
• Lemma 2
• proof
• Proposition 2
• proof
• Lemma 2