Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models

TL;DR

Spatial457 tackles the lack of 6D spatial reasoning benchmarks for large multimodal systems by introducing a synthetic, unbiased dataset that encodes four core spatial capabilities (multi-object recognition, 2D/3D locations, and 3D orientation) across five difficulty levels and seven question types. It presents a scene rendering pipeline, new 6D questions (3D Relate and collision), and an evaluation protocol using RPDR to quantify factor-wise declines. Experiments show state-of-the-art LMMs struggle with 3D and 6D tasks and reveal attribute biases, with a neural symbolic baseline providing an upper bound. A real-world extension via SUN-RGBD demonstrates larger gaps to human performance, underscoring the need for future benchmark expansion and bias mitigation.

Abstract

Although large multimodal models (LMMs) have demonstrated remarkable capabilities in visual scene interpretation and reasoning, their capacity for complex and precise 3-dimensional spatial reasoning remains uncertain. Existing benchmarks focus predominantly on 2D spatial understanding and lack a framework to comprehensively evaluate 6D spatial reasoning across varying complexities. To address this limitation, we present Spatial457, a scalable and unbiased synthetic dataset designed with 4 key capability for spatial reasoning: multi-object recognition, 2D location, 3D location, and 3D orientation. We develop a cascading evaluation structure, constructing 7 question types across 5 difficulty levels that range from basic single object recognition to our new proposed complex 6D spatial reasoning tasks. We evaluated various large multimodal models (LMMs) on PulseCheck457, observing a general decline in performance as task complexity increases, particularly in 3D reasoning and 6D spatial tasks. To quantify these challenges, we introduce the Relative Performance Dropping Rate (RPDR), highlighting key weaknesses in 3D reasoning capabilities. Leveraging the unbiased attribute design of our dataset, we also uncover prediction biases across different attributes, with similar patterns observed in real-world image settings. The code and data are released in https://github.com/XingruiWang/Spatial457.

Figures (15)

• Figure 1: Overview of the Spatial457 benchmark. We define four core capabilities for spatial reasoning. By incorporating these capabilities step-by-step, the benchmark assesses models across five progressive difficulty levels and seven question types, ranging from single-object recognition to advanced 6D spatial relationships and collision prediction. Our evaluation demonstrates a significant performance drop for more advanced questions and highlights the gap between state-of-the-art (SoTA) models and human performance in complex spatial reasoning VQA tasks.
• Figure 2: The distribution of 3D poses in representative 6D datasets (a&b) highlights an imbalance in object orientations. This inspired us to develop a balanced benchmark encompassing all degrees of 3D orientation.
• Figure 3: (a.1-3) Example of the 2D spatial, 3D spatial and collision with images and questions. (b.1-2) shows the new operation programs we generated for the new 6D spatial reasoning questions.
• Figure 4: The distribution of color and pose attributes in the L4-3D-Pose task for GPT-4o and Gemini-Pro1.5. Although the ground truth labels are well balanced, the predicted colors and poses for both models are imbalanced, with "yellow" being the most frequent color and "front" the most frequent pose.
• Figure 5: Example of GPT-4o across different spatial reasoning tasks on the same image with the ground truth answer marked in blue. GPT-4o answers multi-object, 2D spatial, and occlusion tasks correctly (in the green block) but fails on 3D pose and 6D spatial questions (the red block).