MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models

TL;DR

MMComposition introduces a comprehensive, human-annotated benchmark to rigorously evaluate fine-grained compositionality in pre-trained Vision-Language Models across perception, reasoning, and probing tasks. It aggregates 4,342 questions over 13 categories and 4,342 questions to diagnose where current VLMs fall short, including multi-image and indefinite-choice scenarios. The study analyzes 54 VLMs, finds substantial gaps relative to human performance, and identifies key factors—visual encoder design, data volume, and language decoder size—that influence compositionality. The results illuminate concrete failure modes and offer actionable directions for model design and training to close the compositionality gap in VL understanding.

Abstract

The advent of large Vision-Language Models (VLMs) has significantly advanced multimodal understanding, enabling more sophisticated and accurate integration of visual and textual information across various tasks, including image and video captioning, visual question answering, and cross-modal retrieval. Despite VLMs' superior capabilities, researchers lack a comprehensive understanding of their compositionality -- the ability to understand and produce novel combinations of known visual and textual components. Prior benchmarks provide only a relatively rough compositionality evaluation from the perspectives of objects, relations, and attributes while neglecting deeper reasoning about object interactions, counting, and complex compositions. However, compositionality is a critical ability that facilitates coherent reasoning and understanding across modalities for VLMs. To address this limitation, we propose MMCOMPOSITION, a novel human-annotated benchmark for comprehensively and accurately evaluating VLMs' compositionality. Our proposed benchmark serves as a complement to these earlier works. With MMCOMPOSITION, we can quantify and explore the compositionality of the mainstream VLMs. Surprisingly, we find GPT-4o's compositionality inferior to the best open-source model, and we analyze the underlying reasons. Our experimental analysis reveals the limitations of VLMs in fine-grained compositional perception and reasoning, and points to areas for improvement in VLM design and training. Resources available at: https://hanghuacs.github.io/MMComposition/

Figures (15)

• Figure 1: MMComposition comprises 13 categories of high-quality VL composition QA pairs, covering a wide range of complex compositions. In the example, GPT-4o failed to understand the compositional aspects of the visual and textual components, misidentifying a three-story building as a double-decker structure. This misinterpretation highlights the limitations of current VLMs.
• Figure 2: The statistics of 13 distinct categories of QA pairs in MMComposition and some models' performance on each category.
• Figure 3: Interpretable analysis of different VLMs. Green letters indicate correct answers, while red letters represent wrong (predicted) answers.
• Figure 4: Distribution of difficulty levels across the question set, illustrating the challenging nature of tasks.
• Figure 5: Distribution of option counts per question, showing the variety in answer choices provided to evaluate VLMs.