Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation

TL;DR

MathSpatial introduces a unified ecosystem for mathematical spatial reasoning in multimodal LLMs, pairing a perception-isolated benchmark with a large, curated training corpus and a structured reasoning trace scheme (Correlate-Constrain-Infer). The results reveal a substantial gap between humans and current LLMs on spatial tasks ($>95\%$ vs below $60\%$) and demonstrate that structured SRT supervision can improve open-source models while reducing token usage by about $25\%$. The work also shows that reasoning skills learned on MathSpatial transfer to perception-heavy external benchmarks, indicating generalization beyond clean datasets. Overall, MathSpatial provides scalable resources and a principled framework to diagnose, supervise, and advance mathematical spatial reasoning in multimodal systems.

Abstract

Multimodal large language models (MLLMs) have achieved strong performance on perception-oriented tasks, yet their ability to perform mathematical spatial reasoning, defined as the capacity to parse and manipulate two- and three-dimensional relations, remains unclear. Humans easily solve textbook-style spatial reasoning problems with over 95\% accuracy, but we find that most leading MLLMs fail to reach even 60\% on the same tasks. This striking gap highlights spatial reasoning as a fundamental weakness of current models. To investigate this gap, we present MathSpatial, a unified framework for evaluating and improving spatial reasoning in MLLMs. MathSpatial includes three complementary components: (i) MathSpatial-Bench, a benchmark of 2K problems across three categories and eleven subtypes, designed to isolate reasoning difficulty from perceptual noise; (ii) MathSpatial-Corpus, a training dataset of 8K additional problems with verified solutions; and (iii) MathSpatial-SRT, which models reasoning as structured traces composed of three atomic operations--Correlate, Constrain, and Infer. Experiments show that fine-tuning Qwen2.5-VL-7B on MathSpatial achieves competitive accuracy while reducing tokens by 25\%. MathSpatial provides the first large-scale resource that disentangles perception from reasoning, enabling precise measurement and comprehensive understanding of mathematical spatial reasoning in MLLMs.

Figures (9)

• Figure 1: Left: On the proposed MathSpatial-Bench, humans achieve over 95% accuracy while most MLLMs remain below 60%, revealing a significant capability gap. Right: The three core challenges of spatial reasoning and the design of our MathSpatial framework to address them.
• Figure 2: MathSpatial source data construction pipeline: Data Collection and Curation $\rightarrow$ Standardization $\rightarrow$ Geometric Consistency Checking $\rightarrow$ Solution Verification.
• Figure 3: Examples spanning the three categories of MathSpatial.
• Figure 4: Distribution and composition of the MathSpatial-Bench
• Figure 5: Qualitative comparison of model outputs on a sample problem. GPT-4o and Qwen2.5-VL-7B produce free-form reasoning with inconsistencies or errors, MathSpatial-SRT yields a structured reasoning trace based on atomic operations (Correlate, Constrain, Infer), leading to a correct solution.

Theorems & Definitions (2)

• Proposition 1: Normal-form coverage
• Proposition 2: Minimality