Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency

TL;DR

VCBench introduces a comprehensive benchmark for multimodal mathematical reasoning with explicit visual dependencies, addressing the gap in evaluating elementary-level visual-math abilities across multiple images. It aggregates 1,720 problems over six cognitive domains with 6,697 images, and evaluates 26 LVLMs, revealing substantial performance gaps—top models struggle to exceed 50% accuracy on multi-image tasks. The study systematically analyzes single- vs multi-image settings, chain-of-thought effects, and error modalities, showing that vision-centric cross-image reasoning remains a major bottleneck despite advances in LVLMs. The results provide a benchmark-driven roadmap for improving visual perception, cross-image integration, and method robustness toward broader AGI capabilities.

Abstract

Recent advancements in Large Vision-Language Models (LVLMs) have significantly enhanced their ability to integrate visual and linguistic information, achieving near-human proficiency in tasks like object recognition, captioning, and visual question answering. However, current benchmarks typically focus on knowledge-centric evaluations that assess domain-specific expertise, often neglecting the core ability to reason about fundamental mathematical elements and visual concepts. We identify a gap in evaluating elementary-level math problems, which rely on explicit visual dependencies-requiring models to discern, integrate, and reason across multiple images while incorporating commonsense knowledge, all of which are crucial for advancing toward broader AGI capabilities. To address this gap, we introduce VCBENCH, a comprehensive benchmark for multimodal mathematical reasoning with explicit visual dependencies. VCBENCH includes 1,720 problems across six cognitive domains, featuring 6,697 images (averaging 3.9 per question) to ensure multi-image reasoning. We evaluate 26 state-of-the-art LVLMs on VCBENCH, revealing substantial performance disparities, with even the top models unable to exceed 50% accuracy. Our findings highlight the ongoing challenges in visual-mathematical integration and suggest avenues for future LVLM advancements. The project can be found at https://alibaba-damo-academy.github.io/VCBench/.

Figures (10)

• Figure 1: Representative examples from the VCBench, showcasing diverse question types and categories including Space and Location (Direction, Location and Place), Reasoning and Observation (Reasoning and Observe), Time and Calendar (Calendar and Clock), Objects and Motion (Cube and Move), Organization and Pattern (Weight, Organize and Pattern), and Geometry and Shapes (Shape, Quad, Angle, Rectangular and Triangle).
• Figure 2: (a) Overview of the VCBench dataset structure, highlighting its six main categories and associated subcategories, designed to assess multimodal reasoning capabilities of LVLMs. (b) Distribution of question types in the VCBench, illustrating the relative frequency across different visual reasoning subcategories
• Figure 3: Comparative performance (%) of six various prominent LVLMs across six categories: Time and Calendar (TC), Space and Location (SL), Geometry and Shapes (GS), Objects and Motion (OM), Reasoning and Observation (RO), and Organization and Pattern (OP).
• Figure 4: Comparative evaluation of various LVLMs under Multi-Image and Single-Image settings for the same question. The letters (A, B, C, D) indicate models' predictions, with correct answers marked in green and incorrect answers in red.
• Figure 5: A comparison of error distributions among three model, GPT-4o, Gemini2.0-Flash, and Calude-3.7-Sonnet, across five error categories: visual perception errors, calculation errors, contextual misunderstandings, logical errors, and answer integration errors.