Auto-Comp: An Automated Pipeline for Scalable Compositional Probing of Contrastive Vision-Language Models

TL;DR

Auto-Comp addresses the challenge of evaluating compositional reasoning in Vision-Language Models by automatically generating photorealistic, concept-driven benchmarks with parallel Minimal and Contextual captions for controlled A/B testing. The three-phase pipeline—concept definition, parallel caption generation with automated validation, and hard-negative benchmark creation—enables scalable, artifact-free probing of color and spatial binding. Across 20 VLMs, the study uncovers universal compositional failures, revealing that visio-linguistic context helps spatial reasoning but can hinder local attribute binding, particularly under low-entropy distractors. The work provides a practical benchmarking framework, releases Auto-Comp-CP, and demonstrates generalization to new compositional concepts, offering a path toward more robust visuo-linguistic understanding.

Abstract

Modern Vision-Language Models (VLMs) exhibit a critical flaw in compositional reasoning, often confusing "a red cube and a blue sphere" with "a blue cube and a red sphere". Disentangling the visual and linguistic roots of these failures is a fundamental challenge for robust evaluation. To enable fine-grained, controllable analysis, we introduce Auto-Comp, a fully automated and synthetic pipeline for generating scalable benchmarks. Its controllable nature is key to dissecting and isolating different reasoning skills. Auto-Comp generates paired images from Minimal (e.g., "a monitor to the left of a bicycle on a white background") and LLM-generated Contextual captions (e.g., "In a brightly lit photography studio, a monitor is positioned to the left of a bicycle"), allowing a controlled A/B test to disentangle core binding ability from visio-linguistic complexity. Our evaluation of 20 VLMs on novel benchmarks for color binding and spatial relations reveals universal compositional failures in both CLIP and SigLIP model families. Crucially, our novel "Confusion Benchmark" reveals a deeper flaw beyond simple attribute swaps: models are highly susceptible to low-entropy distractors (e.g., repeated objects or colors), demonstrating their compositional failures extend beyond known bag-of-words limitations. we uncover a surprising trade-off: visio-linguistic context, which provides global scene cues, aids spatial reasoning but simultaneously hinders local attribute binding by introducing visual clutter. We release the Auto-Comp pipeline to facilitate future benchmark creation, alongside all our generated benchmarks (https://huggingface.co/AutoComp).

Figures (4)

• Figure 1: The Auto-Comp pipeline for automated benchmark creation. (1) Caption Generation: Concept-driven generation of template-based Minimal and LLM-based Contextual captions. (2) Image Synthesis & Validation: A text-to-image model generates visuals, which are then validated using GroundingDINO and VLM checks. (3) Hard Negative Generation: Validated pairs are systematically altered by swapping attributes to create challenging compositional test sets.
• Figure 2: Examples of positive captions-image pairs from the Auto-Comp-CP benchmarks.
• Figure 3: Distribution of validated samples per attribute in our final datasets, summed across N=2 and N=3 tasks. (Left) Total samples per color in the Color Benchmark. (Right) Total samples per relation in the Position Benchmark.
• Figure 4: Qualitative examples from the newly generated Shape-Color and Relative Size benchmarks. The Auto-Comp pipeline generalizes to these new domains without architectural changes.