In-Context Learning Improves Compositional Understanding of Vision-Language Models

TL;DR

The paper tackles the challenge of compositional understanding in vision-language models (VLMs) by comparing contrastive and generative approaches and introducing an in-context learning (ICL) prompting framework. By using synthetic captions/images generated via GPT-4o and real COCO data as few-shot demonstrations, the authors show that ICL can enhance compositional reasoning across benchmarks such as ARO, Winoground, and SugarCrepe without updating model parameters. Key findings include robust improvements from both synthetic and real demonstrations and nuanced advantages of generative versus contrastive architectures depending on the task (attributes/relations vs. ordering). The work highlights a practical path to boost multimodal compositionality through task-specific, few-shot prompting, and points to future directions like patch-level pretraining and neurosymbolic grounding to further improve reasoning capabilities.

Abstract

Vision-Language Models (VLMs) have shown remarkable capabilities in a large number of downstream tasks. Nonetheless, compositional image understanding remains a rather difficult task due to the object bias present in training data. In this work, we investigate the reasons for such a lack of capability by performing an extensive bench-marking of compositional understanding in VLMs. We compare contrastive models with generative ones and analyze their differences in architecture, pre-training data, and training tasks and losses. Furthermore, we leverage In-Context Learning (ICL) as a way to improve the ability of VLMs to perform more complex reasoning and understanding given an image. Our extensive experiments demonstrate that our proposed approach outperforms baseline models across multiple compositional understanding datasets.

Figures (3)

• Figure 1: Our in-context learning pipeline for compositional understanding of VLMs. We instruct GPT-4o to generate a caption consisting of a few named objects such that they are compositionally defined. Using this positive caption we then instruct GPT-4o to generate an image and a negative caption that compositionally distorts the meaning. We feed these synthetic captions and images as few-shot examples to the VLMs. Afterwards, we instruct the model to predict between correct and wrong captions for images of compositional reasoning benchmarks while also using in-context learning prompting.
• Figure 2: Few-shot samples. First row: The images and captions are synthetically generated with GPT-4o as seen in Figure \ref{['fig:image_gen_pipeline']}. Second row: Images are manually captioned and retrieved from COCO dataset lin2015microsoftcoco. We use these images to instill an understanding of the task within the generative models in a few-shot manner.
• Figure 1.1: Compositional reasoning examples from Winoground thrush2022winoground, showing the close similarity between the pairs of images and text.