SpatialViz-Bench: An MLLM Benchmark for Spatial Visualization

TL;DR

SpatialViz-Bench formalizes a cognitive-science grounded benchmark to assess spatial visualization in multimodal LLMs. It decomposes spatial reasoning into four sub-abilities, implements 12 procedurally generated tasks (1,180 problems), and evaluates 33 MLLMs under zero-shot prompts with and without Chain-of-Thought. The study reveals that current models struggle with perceptual and transformation aspects rather than high-level reasoning, with large performance gaps to human ability and mixed CoT effects. Public data and evaluation code accompany the benchmark to enable ongoing expansion and diagnostic analysis.

Abstract

Humans can directly imagine and manipulate visual images in their minds, a capability known as spatial visualization. While multi-modal Large Language Models (MLLMs) support imagination-based reasoning, spatial visualization remains insufficiently evaluated, typically embedded within broader mathematical and logical assessments. Existing evaluations often rely on IQ tests or math competitions that may overlap with training data, compromising assessment reliability. To this end, we introduce SpatialViz-Bench, a comprehensive multi-modal benchmark for spatial visualization with 12 tasks across 4 sub-abilities, comprising 1,180 automatically generated problems. Our evaluation of 33 state-of-the-art MLLMs not only reveals wide performance variations and demonstrates the benchmark's strong discriminative power, but also uncovers counter-intuitive findings: models show difficulty perception misaligned with human intuition, exhibit dramatic 2Dto-3D performance cliffs, default to formulaic derivation over visualization, and paradoxically suffer performance degradation from Chain-of-Thought prompting in open-source models. Through statistical and qualitative analysis of error types, SpatialViz-Bench demonstrates that state-of-the-art MLLMs continue to exhibit deficiencies in spatial visualization tasks, thereby addressing a significant lacuna in the field. The benchmark data and evaluation code are publicly available.

Figures (33)

• Figure 1: Overview of SpatialViz-Benchmark. The upper panel illustrates the two-phase process of spatial visualization: perceiving visible cues to infer unseen relationships via iterative mental visualization and memorization. The lower-left radar chart displays the zero-shot accuracy of various models, revealing the capability gaps against human performance and random chance. The lower-right table compares our task coverage against other benchmarks.
• Figure 2: The overview of SpatialViz-Bench. SpatialViz-Bench evaluates 4 spatial sub-abilities, mental rotation, mental folding, visual penetration, and mental animation, via 3 tasks each (12 tasks total). Each task has 2–3 difficulty levels of 40–50 cases, yielding 1,180 question–answer pairs.
• Figure 3: The programmatic generation pipeline of a data instance. We constructed the dataset using an programmatic generation system that integrates Python with FreeCAD, enabling precise control of difficulty, systematic generation of distractor options, and programmatic recording of explanations for incorrect choices.
• Figure 4: Comparison of error type distributions, with chart (a) showing the overall breakdown and charts (b-e) detailing results for specific MLLMs: (b) Gemini-2.5, (c) o1, (d) Qwen2.5-VL-72B and (e) Qwen2.5-VL-7B. Errors are classified into six categories: Perceptual, Spatial Transformation, Methodological, Instruction Following, Spatial Memorization, and Calculation & Reasoning.
• Figure 5: Case study of Gemini-2.5-pro's reasoning in different tasks.