Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MentalBlackboard: Evaluating Spatial Visualization via Mathematical Transformations

Nilay Yilmaz, Maitreya Patel, Naga Sai Abhiram Kusumba, Yixuan He, Yezhou Yang

TL;DR

MentalBlackboard introduces an open-ended spatial visualization benchmark for Vision-Language Models based on Paper Folding Test transformations, implemented as prediction and planning tasks with symmetry and rotation. The authors build a large-scale data pipeline that generates dynamic 3D folding animations and provides 2D image and text representations, enabling detailed error analysis beyond mere accuracy. Across video, image, and text modalities, current models show limited spatial reasoning, achieving up to 25% exact-match on prediction and around 10% on planning, with rotation and symmetry notably degrading performance. The work highlights significant gaps in sequential reasoning, visuospatial working memory, and physical awareness in VLMs, while offering a scalable framework and insights to drive advances in spatial visualization for embodied AI and robotics.

Abstract

Spatial visualization is the mental ability to imagine, transform, and manipulate the spatial characteristics of objects and actions. This intelligence is a part of human cognition where actions and perception are connected on a mental level. To explore whether state-of-the-art Vision-Language Models (VLMs) exhibit this ability, we develop MentalBlackboard, an open-ended spatial visualization benchmark for Paper Folding and Hole Punching tests within two core tasks: prediction and planning. Our prediction experiments reveal that models struggle with applying symmetrical transformations, even when they predict the sequence of unfolding steps correctly. Also, rotations introduce a significant challenge to the physical situational awareness for models. The planning task reveals limitations of models in analyzing symmetrical relationships and in implementing the multi-stage symmetry process, with Claude Opus 4.1 achieving the highest planning score at an accuracy of 10\%. The top-performing model, o3, attains a peak performance of 71.6\% on the generalization task, which does not require spatial visualization but transfers spatial data; however, it achieves only 25\% accuracy on text-based prediction tasks.

MentalBlackboard: Evaluating Spatial Visualization via Mathematical Transformations

TL;DR

MentalBlackboard introduces an open-ended spatial visualization benchmark for Vision-Language Models based on Paper Folding Test transformations, implemented as prediction and planning tasks with symmetry and rotation. The authors build a large-scale data pipeline that generates dynamic 3D folding animations and provides 2D image and text representations, enabling detailed error analysis beyond mere accuracy. Across video, image, and text modalities, current models show limited spatial reasoning, achieving up to 25% exact-match on prediction and around 10% on planning, with rotation and symmetry notably degrading performance. The work highlights significant gaps in sequential reasoning, visuospatial working memory, and physical awareness in VLMs, while offering a scalable framework and insights to drive advances in spatial visualization for embodied AI and robotics.

Abstract

Spatial visualization is the mental ability to imagine, transform, and manipulate the spatial characteristics of objects and actions. This intelligence is a part of human cognition where actions and perception are connected on a mental level. To explore whether state-of-the-art Vision-Language Models (VLMs) exhibit this ability, we develop MentalBlackboard, an open-ended spatial visualization benchmark for Paper Folding and Hole Punching tests within two core tasks: prediction and planning. Our prediction experiments reveal that models struggle with applying symmetrical transformations, even when they predict the sequence of unfolding steps correctly. Also, rotations introduce a significant challenge to the physical situational awareness for models. The planning task reveals limitations of models in analyzing symmetrical relationships and in implementing the multi-stage symmetry process, with Claude Opus 4.1 achieving the highest planning score at an accuracy of 10\%. The top-performing model, o3, attains a peak performance of 71.6\% on the generalization task, which does not require spatial visualization but transfers spatial data; however, it achieves only 25\% accuracy on text-based prediction tasks.
Paper Structure (33 sections, 1 equation, 38 figures, 17 tables)

This paper contains 33 sections, 1 equation, 38 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MentalBlackboard
  4. Benchmark Construction
  5. Environment Setup.
  6. Dataset Configuration.
  7. Dataset Generation.
  8. 2D & Text Mapping.
  9. Tasks of MentalBlackboard
  10. Evaluation Setup
  11. Evaluation Results
  12. Prediction
  13. Planning
  14. Discussion
  15. Do VLMs Transfer Spatial Information?
  16. ...and 18 more sections

Figures (38)

  • Figure 1: The MentalBlackboard creation pipeline, which consists of environment setup, dataset configuration, dataset generation, and 2D and text mapping.
  • Figure 2: Examples of a prediction task and a planning task.
  • Figure 3: Group-wise performance loss of o3 models across task modalities under exact hole and unfolding-step matching.
  • Figure 4: Partial accuracy on planning task per field
  • Figure 5: An example of a generalization task employs a visual analogical reasoning framework, which requires making an inference from the first scenario to identify the missing part in the second scenario
  • ...and 33 more figures