HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models

Abstract

Achieving human-like spatial intelligence for vision-language models (VLMs) requires inferring 3D structures from 2D observations, recognizing object properties and relations in 3D space, and performing high-level spatial reasoning. In this paper, we propose a principled hierarchical framework that decomposes the learning of 3D spatial understanding in VLMs into four progressively complex levels, from geometric perception to abstract spatial reasoning. Guided by this framework, we construct an automated pipeline that processes approximately 5M images with over 45M objects to generate 3D spatial VQA pairs across diverse tasks and scenes for VLM supervised fine-tuning. We also develop an RGB-D VLM incorporating metric-scale point maps as auxiliary inputs to further enhance spatial understanding. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on multiple spatial understanding and reasoning benchmarks, surpassing specialized spatial models and large proprietary systems such as Gemini-2.5-pro and GPT-5. Moreover, our analysis reveals clear dependencies among hierarchical task levels, offering new insights into how multi-level task design facilitates the emergence of 3D spatial intelligence.

Figures (19)

• Figure 1: Overview of our approach. Left: Data construction pipeline which generates spatial-related VQA pairs from either in-the-wild images or existing data with 3D annotations. Right: Hierarchical spatial understanding task taxonomy with representative QA pairs.
• Figure 2: Model architecture of our VLM, which integrates metric-scale 3D point map as auxiliary input.
• Figure 3: Examples of spatial VQA data constructed using our method, covering different task levels.
• Figure 4: Examples of spatial VQA data constructed using our method, covering different task levels.
• Figure 5: Examples of our model’s responses on unseen images.