Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lens
Ziliang Chen, Tianang Xiao, Jusheng Zhang, Yongsen Zheng, Xipeng Chen
TL;DR
The paper tackles CLIP's persistent failures in vision-language compositionality by introducing a token-level causal representation learning (CRL) framework built on a language-token SCM. It extends block identifiability to tokenized text, proving that the modal-invariant latent $z_{\rm inv}$ can be recovered under both sentence- and token-level data generation, while also revealing composition nonidentifiability via pseudo-optimal encoders $g^{**}$ that align yet ignore token-level swaps, replacements, and additions. The authors connect language-side nonidentifiability to visual modality gaps and show how iterated composition operators can produce increasingly hard negatives, motivating advanced negative mining strategies. Empirically, they demonstrate that token-level perturbations reproduce many benchmark hard negatives and improve CLIP-based models when used in training, thereby bridging theory to practice and offering practical routes to bolster compositional generalization in multimodal models.
Abstract
Contrastive Language-Image Pre-training (CLIP) delivers strong cross modal generalization by aligning images and texts in a shared embedding space, yet it persistently fails at compositional reasoning over objects, attributes, and relations often behaving like a bag-of-words matcher. Prior causal accounts typically model text as a single vector, obscuring token-level structure and leaving core phenomena-such as prompt sensitivity and failures on hard negatives unexplained. We address this gap with a token-aware causal representation learning (CRL) framework grounded in a sequential, language-token SCM. Our theory extends block identifiability to tokenized text, proving that CLIP's contrastive objective can recover the modal-invariant latent variable under both sentence-level and token-level SCMs. Crucially, token granularity yields the first principled explanation of CLIP's compositional brittleness: composition nonidentifiability. We show the existence of pseudo-optimal text encoders that achieve perfect modal-invariant alignment yet are provably insensitive to SWAP, REPLACE, and ADD operations over atomic concepts, thereby failing to distinguish correct captions from hard negatives despite optimizing the same training objective as true-optimal encoders. The analysis further links language-side nonidentifiability to visual-side failures via the modality gap and shows how iterated composition operators compound hardness, motivating improved negative mining strategies.