Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs

TL;DR

This work reveals that mathematical diagram understanding in Multimodal LLMs is dominated by perceptual deficits, coining the term Math Blind. It introduces MATHE METRIC to isolate perceptual tasks and GEO METRIC to train structure-aware geometric perception, showing that perception improvements translate into downstream reasoning gains. Across 20 MLLMs, enhanced geometric perception yields a +79% grounding boost and 3–4% cross-suite gains on public benchmarks without extra chain-of-thought data, underscoring the link between low-level perception and high-level math reasoning. The findings advocate for graph-like representations of diagram primitives and careful data design to bridge perception with robust mathematical inference in AI systems.

Abstract

Diagrams represent a form of visual language that encodes abstract concepts and relationships through structured symbols and their spatial arrangements. Unlike natural images, they are inherently symbolic, and entirely artificial. They thus pose unique challenges for Multimodal Large Language Models (MLLMs) distinct from natural image processing. Recent studies have shown that MLLMs often exhibit flawed reasoning and hallucinations when handling diagram inputs. We investigate here whether these limitations stem from shortcomings in the models' ability to interpret diagrams themselves. To this end, we develop a diagnostic test suite that isolates perception from reasoning. Our systematic evaluation reveals that MLLMs perform poorly on basic perceptual tasks, e.g., shape classification, object counting, relationship identification, and object grounding, with near-zero accuracy on fine-grained grounding. Further analysis shows that weak diagram perception leads to "blind faith in text", where models rely on textual shortcuts rather than visual understanding (that is, they are Math Blind). We hypothesize that enabling models to capture the inherent structural properties of diagrams, represented as graphs of primitives and their interrelationships, is essential for improving diagram understanding. Experiments with 7B and 32B MLLMs validate this assumption, with models trained on such representations achieving a +79% gain on the grounding task. Crucially, these gains transfer to reasoning, achieving 3-4% cross-suite improvements on three public benchmarks even without additional chain-of-thought reasoning data. Our findings demonstrate that low-level perception supports faithful high-level reasoning in mathematical MLLMs. We provide both methodological frameworks and empirical evidence to guide future research in this direction.

Figures (28)

• Figure 1: Performance on MATHE METRIC reveals that diagram interpretation is challenging for MLLMs, particularly in fine-grained grounding tasks that require precise spatial localization. SVE-Math-DeepSeek$^+$-7B, trained on GEO METRIC, significantly outperforms comparators, validating the structure-aware geometric data design.
• Figure 2: Illustration of diagram-caption alignment training datasets, w.r.t., AutoGeo, MAVIS, and our GEO METRIC (Fig. \ref{['fig:intro_subfig1']}) with shapes in green, relationships in yellow, box locations in blue, and ambiguity in red. Fig. \ref{['fig:intro_subfig2']} demonstrates a positive correlation between low-level perception and high-level reasoning tasks, evaluated on MATHE METRIC and MathVerse. Clear diagram perception leads to substantial improvements in mathematical reasoning performance.
• Figure 3: Sampled MATHE METRIC examples from plane geometry w.r.t. each question-answer (Q&A) type, covering shape classification, object counting, relationship identification, and object grounding (from left to right). The green dotted bounding box is shown for illustration purposes only and are not provided as input to the models.
• Figure 4: We synthesize geometric figures by randomly sampling elements from the geometric shape pool and relationship pool, ensuring consistency through a verifier that enforces logical constraints based on manually designed rules, fundamental mathematical principles, and prerequisite points. All visual elements are structured and saved in JSON format. Images are rendered using the Matplotlib package, and corresponding Q&A pairs are generated using a template-based pipeline.
• Figure 5: We evaluate five key factors influencing MLLM perception: object count (# obejct), visual quality, visual distractors, textual distractors, and Chain-of-Thought (CoT) reasoning, with close-up results shown in Tabs \ref{['tab:visfactor']}-\ref{['tab:textfactor']} and Fig. \ref{['fig:erro']}.