Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions

TL;DR

This work tackles the poor compositional reasoning of vision-language models trained with standard contrastive objectives by focusing on the text encoder. It introduces READ, a fine-tuning method that adds token-level reconstruction and sentence-level alignment to the CLIP objective, encouraging encoding of word relations and paraphrase semantics. READ-CLIP achieves state-of-the-art performance across five compositional benchmarks and improves existing CLIP variants, with analyses showing complementary benefits and robustness to paraphrase quality. The approach offers a practical path to more relationally aware multimodal representations and can be applied to existing VLMs with minimal overhead.

Abstract

Despite recent advances, vision-language models trained with standard contrastive objectives still struggle with compositional reasoning -- the ability to understand structured relationships between visual and linguistic elements. This shortcoming is largely due to the tendency of the text encoder to focus on individual words rather than their relations, a limitation reinforced by contrastive training that primarily aligns words with visual objects. In this paper, we introduce REconstruction and Alignment of text Descriptions (READ), a fine-tuning method designed to enhance compositional reasoning by adding two auxiliary objectives to the contrastive learning: (1) a token-level reconstruction objective, where a frozen pre-trained decoder reconstructs alternative captions based on the embedding of the original caption; and (2) a sentence-level alignment objective, which explicitly aligns paraphrased sentences in the embedding space. We show that READ-CLIP, a model derived by applying the READ method to the pre-trained CLIP model, achieves the state-of-the-art performance across five major compositional reasoning benchmarks, outperforming the strongest conventional fine-tuning baseline by up to 4.1%. Furthermore, applying the READ to existing CLIP variants (including NegCLIP and FSC-CLIP) also improves performance on these benchmarks. Quantitative and qualitative analyses reveal that our proposed objectives -- reconstruction and alignment -- offer complementary benefits: the former encourages the encoder to capture relationships between words within a caption, while the latter ensures consistent representations for paraphrases expressed with different wording.

Figures (8)

• Figure 1: Overview of the training objectives used in READ-CLIP (left), a VLM that applies REconstruction and Alignment of text Descriptions (READ) method to the pretrained CLIP model radford2021learning. READ is our proposed fine-tuning method that enhances compositional reasoning in VLMs by augmenting contrastive learning with two auxiliary objectives. The auxiliary objectives consist of two components: token-level reconstruction and sentence-level alignment. The performance of READ-CLIP (right) on compositional reasoning benchmarks demonstrates that it consistently outperforms conventional state-of-the-art methods across diverse aspects of compositional reasoning.
• Figure 2: Illustration of our proposed auxiliary objectives of REconstruction and Alignment of text Description (READ) method. Given pairs of captions -- an original and its paraphrase -- that share a common meaning, the token-level reconstruction (1) trains the text encoder to produce embeddings from the original caption such that a frozen pre-trained decoder can reconstruct each token of the paraphrased caption. This reconstruction encourages the encoder to capture relationships between words within a caption, which are critical for reconstructing its paraphrase. In contrast, the sentence-level alignment (2) aligns the pair of captions in the embedding space. This alignment encourages the encoder to capture underlying semantic relationships across the paraphrased captions.
• Figure 3: Representative examples illustrating how applying the reconstruction loss affects caption rankings based on image-caption cosine similarity on WhatsUp kamath-etal-2023-whats and SugarCrepe hsieh2024sugarcrepe. Positive (✓) and negative captions (✗) are shown with their rankings based on image-caption cosine similarity.
• Figure 4: Representative examples from each category (swap and replace) of SugarCrepe++ dumpala2024sugarcrepepp dataset, showing how applying the alignment loss improves the ranking of positive captions. In Sugarcrepe++, each image is paired with two positive captions (✓) that are worded differently. In ITT (image-to-text) evaluation, a prediction is considered accurate if both positive captions are ranked higher than all negatives based on image-caption cosine similarity.
• Figure 5: Comparison of reconstructing a paraphrased caption versus the original one, measured by the performance of trained encoder on SugarCrepe++ dumpala2024sugarcrepepp TOT (text-to-text) benchmark, for various training epochs. In the TOT evaluation, average accuracy (left) is computed by checking whether the cosine similarity between the positive caption pair is higher than that of any positive–negative pairs. Average similarity (right) measures cosine similarity between caption pairs. Here, Pos1 and Pos2 are positive pairs, while others are negative pairs.