Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View

TL;DR

The paper tackles the problem of poor compositional reasoning in Vision Language Models by introducing a novel game-theoretic framework based on the Harsanyi dividend to quantify how input patterns (objects, relations, attributes) influence text and image encoders, as well as multimodal interactions. It formalizes the framework with $w(\mathcal{S}|\mathcal{N}) = \sum_{\mathcal{S}' \subseteq \mathcal{S}} (-1)^{|\mathcal{S}'| - |\mathcal{S}|} v(\mathcal{S}')$ and uses a reward function $v(\cdot)$ (e.g., cosine similarity) to derive sensitivity metrics $Q_R$, $Q_A$, $Q_O$, $Q_{R\&O}$, $Q_{A\&O}$ for text encoders, evaluating five state-of-the-art VLMs across benchmarks like ARO, SUGARCREPE, VL-CheckList, and Visual Genome Relation. The findings show text encoders exhibit strong, human-aligned compositional sensitivities while image encoders are weaker, and cross-modal analyses reveal a lack of mutually matching compositional knowledge between text and image streams. These insights offer concrete directions to improve visual encoders and to better align cross-modal compositional understanding, potentially via dedicated modules or auxiliary losses informed by the proposed diagnostics.

Abstract

Compositional reasoning capabilities are usually considered as fundamental skills to characterize human perception. Recent studies show that current Vision Language Models (VLMs) surprisingly lack sufficient knowledge with respect to such capabilities. To this end, we propose to thoroughly diagnose the composition representations encoded by VLMs, systematically revealing the potential cause for this weakness. Specifically, we propose evaluation methods from a novel game-theoretic view to assess the vulnerability of VLMs on different aspects of compositional understanding, e.g., relations and attributes. Extensive experimental results demonstrate and validate several insights to understand the incapabilities of VLMs on compositional reasoning, which provide useful and reliable guidance for future studies. The deliverables will be updated at https://vlms-compositionality-gametheory.github.io/.

Figures (8)

• Figure 1: Diagnosing the compositional reasoning capabilities of Vision Language Models (VLMs). In this paper, we systematically analyze the potential causes for the poor compositional performance of VLMs from each unimodal separately and then multimodal jointly. In this way, three insights are obtained and validated correspondingly. Please see Section \ref{['sec:intro']} for detailed elaborations.
• Figure 2: Evaluating the sensitivities of text encoders of VLMs to the changes of textual patterns. Specifically, given captions $\mathcal{T}_1$ and $\mathcal{T}_2$ with object words being swapped, we design $Q_O$, $Q_R$ and $Q_{R\&O}$ to assess whether text encoders react correctly to the fine-grained changes of compositionality. In this case, $Q_{R\&O}$, which measures the interaction changes between object words and relation words, should be of a greater value than $Q_O$ and $Q_R$. Here $\text{T}_1 \cdot \text{I}_1$ represents the cosine similarity between the normalized text embedding $\text{T}_1$ and normalized image embedding $\text{I}_1$.
• Figure 3: The Pearson correlation coefficients $\rho(\mathcal{X^T}, \mathcal{Y^T})$ between the reward differences $\mathcal{X^T}$ and interaction effect differences on the Visual Genome Relation dataset, i.e., $\mathcal{Y}^\mathcal{T}_R$, $\mathcal{Y}^\mathcal{T}_O$ and $\mathcal{Y}^\mathcal{T}_{R \& O}$. Each point represents a data sample containing two captions and one image. Results show that the reward differences between the two captions were mainly related to the interaction changes of object words and relation words, demonstrating that text encoders of VLMs reacted correctly to the textual compositional differences in this dataset.
• Figure 4: Evaluating the sensitivities of image encoders of VLMs to the changes of visual patterns. Specifically, given images $\mathcal{I}_1$ and $\mathcal{I}_2$ with object relations being altered, we design $D_{O_1}$, $D_{O_2}$ and $D_{O_1\&O_2}$ to assess whether image encoders react correctly to the fine-grained changes of visual compositionality. In this case, $D_{O_1\&O_2}$, which measures the relation changes between objects, should be of a greater value than $D_{O_1}$ and $D_{O_2}$.
• Figure 5: Evaluating whether image encoders and text encoders of VLMs possess mutually matching compositional knowledge with modified sensitivity metrics. Specifically, given image-text pairs $\{\mathcal{I}_1,\mathcal{T}_1\}$ and $\{\mathcal{I}_2, \mathcal{T}_2\}$ sharing minimal differences of object relations, we design ${Q_{\mathcal{T}:R\&O \xrightarrow{}\mathcal{I}:O_1}}$, ${Q_{\mathcal{T}:R\&O \xrightarrow{}\mathcal{I}:O_2}}$ and ${Q_{\mathcal{T}:R\&O \xrightarrow{}\mathcal{I}:O_1\& O_2}}$ to assess whether image encoders obtain the corresponding compositional knowledge for text encoders. Besides, we also design $D_{\mathcal{I}:O_1\&O_2 \xrightarrow{}\mathcal{T}:O}$, $D_{\mathcal{I}:O_1\&O_2 \xrightarrow{}\mathcal{T}:R}$ and $D_{\mathcal{I}:O_1\&O_2 \xrightarrow{}\mathcal{T}:R\&O}$ to assess whether text encoders obtain the corresponding compositional knowledge for image encoders. Please zoom in for better visualization.