iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs

TL;DR

iVISPAR tackles the challenge of evaluating spatial reasoning in Vision-Language Models by introducing an interactive, multimodal benchmark built around the Sliding Geom Puzzle. It enables agents to operate in 3D, 2D, and text modalities, with procedurally generated tasks of scalable complexity and benchmarking against optimal A* solutions and human baselines. The results reveal a persistent gap between current VLMs and human spatial cognition, especially in 3D settings, while 2D inputs often yield the strongest performance for many models. Overall, iVISPAR provides a rigorous, reproducible platform for advancing grounded spatial reasoning in VLMs and guides future extensions into broader scene understanding and geometric transformations.

Abstract

Vision-Language Models (VLMs) are known to struggle with spatial reasoning and visual alignment. To help overcome these limitations, we introduce iVISPAR, an interactive multimodal benchmark designed to evaluate the spatial reasoning capabilities of VLMs acting as agents. \mbox{iVISPAR} is based on a variant of the sliding tile puzzle, a classic problem that demands logical planning, spatial awareness, and multi-step reasoning. The benchmark supports visual 3D, 2D, and text-based input modalities, enabling comprehensive assessments of VLMs' planning and reasoning skills. We evaluate a broad suite of state-of-the-art open-source and closed-source VLMs, comparing their performance while also providing optimal path solutions and a human baseline to assess the task's complexity and feasibility for humans. Results indicate that while VLMs perform better on 2D tasks compared to 3D or text-based settings, they struggle with complex spatial configurations and consistently fall short of human performance, illustrating the persistent challenge of visual alignment. This underscores critical gaps in current VLM capabilities, highlighting their limitations in achieving human-level cognition. Project website: https://microcosm.ai/ivispar

Figures (12)

• Figure 1: VLMs' success rates of completed games over 900 episodes across vision 3D, vision 2D, and text.
• Figure 2: Example of VLMs' observations for a state (blue) and the goal (green) at each step during an episode of the Sliding Geom Puzzle environment, on a 4$\times$4 board with 10 geoms and an optimal path length of 2. Left to right, each tested modality: vision 3D, vision 2D, and text-based representation. For more examples, see Appendix \ref{['sec:obs_scaling']}
• Figure 3: Depiction of the interaction flow between VLM agents and the iVISPAR simulator with a progression through an episode with the shortest path solution of 4 steps being solved by prompted actions from a VLM agent. For a full example of an episode progression, see Appendix \ref{['sec:episode_progression']}.
• Figure 4: VLM evaluation on 900 episodes per model across all three modalities, with 95% confidence intervals. Baseline comparisons for human performance and random moves are shown. Top: VLMs' success rates of episodes completed with higher values denoting better performance. Bottom: VLMs' mean step deviation from the optimal path with lower values denoting better performance. Full numerical results are provided in Appendix \ref{['sec:detailed_results']}
• Figure 5: Error patterns showing average action counts per episode during SGP interaction (top) and average geoms per episode for the board state inference auxiliary task (bottom), both averaged across modalities (see Sections \ref{['results']} and \ref{['sec:auxiliary_task']}), each aggregated across modalities. Full numerical results are provided in Appendix \ref{['sec:detailed_results']}.