MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?

TL;DR

MathSight introduces a university-level multimodal benchmark with multiple visual variants and a text-only condition to isolate the true contribution of visual information in mathematical reasoning. Across diverse state-of-the-art VLMs, results show that visual input provides limited advantage as problems become harder, and in many cases text-only reasoning matches or surpasses multimodal variants, indicating models rely heavily on linguistic priors rather than genuine vision-grounded understanding. The work also proposes a suite of logical-consistency metrics (Token Confidence, Sliding-window Group Metrics, GOM, GSD, GCV) to evaluate reasoning robustness, and analyzes category- and size-related effects to reveal nuanced patterns of visual dependence. Overall, MathSight highlights a fundamental gap between apparent multimodal performance and true vision-grounded reasoning, underscoring the need for benchmarks and models that foster genuine visual understanding in mathematical tasks.

Abstract

Recent advances in Vision-Language Models (VLMs) have achieved impressive progress in multimodal mathematical reasoning. Yet, how much visual information truly contributes to reasoning remains unclear. Existing benchmarks report strong overall performance but seldom isolate the role of the image modality, leaving open whether VLMs genuinely leverage visual understanding or merely depend on linguistic priors. To address this, we present MathSight, a university-level multimodal mathematical reasoning benchmark designed to disentangle and quantify the effect of visual input. Each problem includes multiple visual variants -- original, hand-drawn, photo-captured -- and a text-only condition for controlled comparison. Experiments on state-of-the-art VLMs reveal a consistent trend: the contribution of visual information diminishes with increasing problem difficulty. Remarkably, Qwen3-VL without any image input surpasses both its multimodal variants and GPT-5, underscoring the need for benchmarks like MathSight to advance genuine vision-grounded reasoning in future models.

Figures (15)

• Figure 1: Motivation behind MathSight. Our main motivation is to investigate how different image versions affect the performance of VLMs on university-level and beyond mathematical reasoning problems. Since many existing studies focus on university-level (and below) math, as shown in Table 1, we deliberately curated university-level questions when constructing our dataset.
• Figure 2: Box plots of group coefficient of variation (group_CV) across different models and $k$ settings. Each box covers the 95% central region of group_CV values, the horizontal line inside the box marks the median group_CV, and the points outside indicate a small number of outliers. Left: text-only model with $k=5$. Middle: text-only model with $k=20$. Right: multimodal GPT-4o variants with $k=20$.
• Figure 3: Evaluation results of Vision-Language Models (VLMs) on images of different sizes.
• Figure 4: The result comparison between text-only and different multimodal inputs.
• Figure 5: Overview (Categories and examples) of MathSight.