Text encoders bottleneck compositionality in contrastive vision-language models

TL;DR

The paper interrogates how well single-vector text encodings in vision-language models preserve linguistic compositionality. By constructing CompPrompts and ControlledImCaps, the authors quantify information loss via a text-only recovery probe and demonstrate that even strong encoders struggle with spatial, temporal, negation, and order-sensitive aspects, though some encoders fare better than others. A PoC T5-based decoder shows reconstructability is possible, and text-recoverability correlates with, but does not fully predict, multimodal performance, highlighting bottlenecks beyond the text encoder. The results motivate exploring reconstruction-aware objectives and richer training signals to improve compositional understanding in VL models, and they provide publicly released datasets and code to support further probing research.

Abstract

Performant vision-language (VL) models like CLIP represent captions using a single vector. How much information about language is lost in this bottleneck? We first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture (e.g., single object, to object+property, to multiple interacting objects). Then, we train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL models. This approach does not require images, allowing us to test on a broader range of scenes compared to prior work. We find that: 1) CLIP's text encoder falls short on more compositional inputs, including object relationships, attribute-object association, counting, and negations; 2) some text encoders work significantly better than others; and 3) text-only recovery performance predicts multi-modal matching performance on ControlledImCaps: a new evaluation benchmark we collect and release consisting of fine-grained compositional images and captions. Specifically, our results suggest text-only recoverability is a necessary (but not sufficient) condition for modeling compositional factors in contrastive VL models. We release our datasets and code.

Figures (3)

• Figure 1: We present CompPrompts, a dataset of 18,100 text prompts, and ControlledImCaps, a dataset of 600 image pairs+captions that differ by only one word. The two datasets are grouped by the same set of caption properties, e.g., temporal/spatial relations. Experiments on CompPrompts quantify the information loss of a text encoder; experiments on ControlledImCaps illustrate that information loss correlates with multimodal errors.
• Figure 2: We probe the representations of single-vector text encoders used in popular VL models. Using a corpus of increasingly compositional image captions, CompPrompts, we attempt to generatively decode the original input sentence. Text encoders of popular models like CLIP fail to effectively encode precise aspects of their captions like attribute attachments and object relationships (real examples shown, as in Figure \ref{['fig:teaser1']}).
• Figure 3: Each attribute in ControlledImCaps, with comparable prompts in CompPrompts and an example.