MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs

TL;DR

This work tackles the challenge of 3D spatial understanding in multimodal LLMs by introducing the CA-VQA dataset and MM-Spatial. CA-VQA provides high-quality 3D ground truth, multi-view imagery, and metric depth to fine-tune and evaluate 3D reasoning in indoor scenes, while mitigating language priors through blind filtering. MM-Spatial, built on the MM1.5 framework, leverages multi-view inputs, depth cues, and chain-of-thought reasoning to achieve state-of-the-art performance on 3D spatial benchmarks (including CV-Bench and SpatialRGPT-Bench) without sacrificing broader capabilities. The results demonstrate that depth inputs and depth-based tool-use significantly enhance 3D understanding, and that monocular depth estimation can emerge as a learned skill from data alone, signaling strong potential for depth-aware reasoning in MLLMs. This work lays a foundation for robust, generalist 3D perception in AI systems used in robotics, AR/VR, and related applications, and points to future work extending outdoor scenes and further refining 3D grounding capabilities.

Abstract

Multimodal large language models (MLLMs) excel at 2D visual understanding but remain limited in their ability to reason about 3D space. In this work, we leverage large-scale high-quality 3D scene data with open-set annotations to introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data covers diverse spatial tasks including spatial relationship prediction, metric size and distance estimation, and 3D grounding. We show that CA-VQA enables us to train MM-Spatial, a strong generalist MLLM that also achieves state-of-the-art performance on 3D spatial understanding benchmarks, including our own. We show how incorporating metric depth and multi-view inputs (provided in CA-VQA) can further improve 3D understanding, and demonstrate that data alone allows our model to achieve depth perception capabilities comparable to dedicated monocular depth estimation models.

Figures (8)

• Figure 1: (Left) We generate the Cubify Anything VQA (CA-VQA) dataset and benchmark, covering various 1) input signals: single image, metric depth (sensor-based and estimated), multi-frame/-view, and 2) spatial understanding tasks: e.g., relationship prediction, metric estimation, 3D grounding. (Right) We train MM-Spatial, a generalist multimodal LLM that excels at 3D spatial understanding. It supports Chain-of-Thought spatial reasoning involving 2D grounding and depth estimation, and can also leverage depth input via tool-use.
• Figure 2: CA-VQA Data Example. Example of a single sample from our dataset. Each reference frame has between 0-4 multi-view support frames. All frames (reference and support) come with three metric depth maps: Ground truth (FARO laser), ARKit Depth (LiDAR-fused) and Monocular (DepthPro). Each support frame contains the relative pose from the reference image, along with camera intrinsics.
• Figure 3: Example of leveraging depth maps via tool-use. The model predicts the objects' 2D bounding boxes and function calls, receives the tool outputs (which is the median depth value within the box, marked with an $\mathbf{\times}$), and finally reasons about the answer.
• Figure 4: Qualitative Example. We show the predictions of various models on a challenging example from our CA-VQA benchmark. Strong commercial (2a&b) and research models (2c&d) fail. MM-Spatial (1a) is much better, and even more so with CoT enabled (1b), demonstrating our model's strong object grounding (see predicted 2D boxes in the image), depth estimation, and spatial reasoning ability. Accuracy improves further when leveraging ground-truth depth via tool-use (1c), although our CoT model's (1b) predictions are very close to that, for both the intermediate depth values and final answer; monocular estimated depth (1d) is less accurate and yields a worse result.
• Figure 5: CA-VQA Overview. Example QA pairs from our Cubify Anything VQA (CA-VQA) dataset, aiming to unlock object-centric 3D spatial understanding in MLLMs. Using high-quality 3D ground truth annotations from CA-1M lazarow2024cubify, we generate spatial perception questions across a variety of different tasks, e.g., involving relative relationships, metric measurements, and 3D object bounding boxes.