MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs

TL;DR

MLLM-CompBench introduces a comprehensive multimodal benchmark to probe comparative reasoning across eight relativity dimensions using ~39.8K image-pair triplets from fourteen domains. The study systematically evaluates prominent MLLMs (GPT-4V, Gemini-Pro, LLaVA-1.6, VILA-1.5) and analyzes the strengths and weaknesses of current models, finding persistent gaps in existence, spatiality, and quantity relativity, despite strong performance on some state- and emotion-related tasks. The authors also explore two-stage reasoning and fine-tuning, uncovering nuanced effects on specific relativity tasks, and report improvements in newer models (GPT-4o, Gemini1.5-Pro) while noting remaining limitations. By providing extensive data curation, quality controls, and a transparent evaluation protocol, the work establishes a solid foundation for advancing comparative reasoning in MLLMs and guiding future architectural and data-collection strategies.

Abstract

The ability to compare objects, scenes, or situations is crucial for effective decision-making and problem-solving in everyday life. For instance, comparing the freshness of apples enables better choices during grocery shopping while comparing sofa designs helps optimize the aesthetics of our living space. Despite its significance, the comparative capability is largely unexplored in artificial general intelligence (AGI). In this paper, we introduce MLLM-CompBench, a benchmark designed to evaluate the comparative reasoning capability of multimodal large language models (MLLMs). MLLM-CompBench mines and pairs images through visually oriented questions covering eight dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. We curate a collection of around 40K image pairs using metadata from diverse vision datasets and CLIP similarity scores. These image pairs span a broad array of visual domains, including animals, fashion, sports, and both outdoor and indoor scenes. The questions are carefully crafted to discern relative characteristics between two images and are labeled by human annotators for accuracy and relevance. We use MLLM-CompBench to evaluate recent MLLMs, including GPT-4V(ision), Gemini-Pro, and LLaVA-1.6. Our results reveal notable shortcomings in their comparative abilities. We believe MLLM-COMPBENCH not only sheds light on these limitations but also establishes a solid foundation for future enhancements in the comparative capability of MLLMs.

Figures (17)

• Figure 1: MLLM-CompBench offers diverse triplets comprising two images, a question about their relativity, and an answer to cover eight types of relativity (see §\ref{['sec:intro']}). See examples along with predictions of GPT-4V gpt.
• Figure 2: MLLM-CompBench curation pipeline, including data selection, question generation, answer annotation, and verification. We rely on combinations of humans, computer programs, MLLMs (specifically GPT-4V gpt), and CLIP similarity clip to select images and generate questions, based on relativity types and available metadata.
• Figure 3: Error Analysis on CompBench. We observe four types of errors where GPT-4V gpt falls short: (i) differentiating colors between objects and backgrounds, (ii) counting small or distant objects, (iii) identifying objects within crowded scenes, and (iv) recognizing out-of-focus details.
• Figure 4: Annotation Interface for MagicBrush.
• Figure 5: Annotation Interface for Spot-the-diff.