Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning

TL;DR

This work tackles the limited compositional understanding of vision-language models like CLIP by introducing CLIC, a lightweight fine-tuning method that uses concatenated image-caption inputs to generate diverse positives and hard negatives without relying on large LLMs. Through a fused objective—contrastive CLIP loss over multiple positives, a dedicated hard-negative loss, and a uni-modal invariance loss—CLIC improves both lexical and semantic compositionality and yields retrieval gains across architectures and pre-training regimes. Experiments show that CLIC achieves state-of-the-art results on SugarCrepe++ and strengthens image- and text-retrieval benchmarks, while preserving zero-shot classification performance and extending benefits to larger models such as CLIPS and LLaVA when integrated with a CLIC vision encoder. The approach is data-efficient, model-agnostic, and scalable, offering practical improvements for downstream VLM tasks and providing a pathway to more robust multimodal reasoning in large-scale systems.

Abstract

Vision-language models like CLIP have demonstrated remarkable zero-shot capabilities in classification and retrieval. However, these models often struggle with compositional reasoning - the ability to understand the relationships between concepts. A recent benchmark, SugarCrepe++, reveals that previous works on improving compositionality have mainly improved lexical sensitivity but neglected semantic understanding. In addition, downstream retrieval performance often deteriorates, although one would expect that improving compositionality should enhance retrieval. In this work, we introduce CLIC (Compositionally-aware Learning in CLIP), a fine-tuning method based on a novel training technique combining multiple images and their associated captions. CLIC improves compositionality across architectures as well as differently pre-trained CLIP models, both in terms of lexical and semantic understanding, and achieves consistent gains in retrieval performance. This even applies to the recent CLIPS, which achieves SOTA retrieval performance. Nevertheless, the short fine-tuning with CLIC leads to an improvement in retrieval and to the best compositional CLIP model on SugarCrepe++. All our models and code are available at https://clic-compositional-clip.github.io

Figures (12)

• Figure 1: Performance of fine-tuning techniques for improving compositionality compared to the pre-trained CLIP ViT-B/32-model. Previous techniques neither yield consistent improvements on SugarCrepe++ (SC++) dumpala2024sugarcrepe++ nor for the retrieval tasks (R@5 of COCO). CLIC is the only method which shows enhanced compositionality and retrieval performance.
• Figure 2: Data generation scheme for CLIC. For every image, we sample an additional image and concatenate the two. This concatenated image is the input to the model alongside five captions: $p_1$, a concatenation of the first sentence from each image. $p_2$ is a sentence-shuffled version of $p_1$. $p_3$ and $p_4$ are concatenations of two additional sentences from each caption, and $n$ is a hard negative constructed by swapping one word from each sentence of $p_1$.
• Figure 3: Query used for re-captioning for the CogVLM model wang2025cogvlm.
• Figure 5: CLIC improves differently pre-trained CLIP models. We show for varied CLIP models, using CLIC (PixelProse) gives consistent improvements on both compositionality and downstream evaluation benchmarks. Specifically, to the best of our knowledge, CLIPS+CLIC-RedCaps yields SOTA numbers SugarCrepe++ for CLIP-like models.
• Figure 6: Comparing response quality of different LLaVA-1.5-7b versions on random WinoGround validation images. We compare the open-ended prompt (Can you describe this image in detail?) based image captioning of LLaVA with CLIP's vision encoder to that of CLIC. Overall, we believe the responses from CLIC enabled LLaVA to be marginally better than the standard LLaVA, but both models show aspects of hallucinations. Incorrect parts are highlighted and the respective corrections (if available) are also highlighted.