SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models

TL;DR

The paper tackles the lack of situated spatial understanding in 3D-based LLMs by (1) constructing Spartun3D, a large-scale, LLM-generated dataset with agent-centric situations and situated scene graphs, and (2) introducing Spartun3D-LLM, a 3D-LMM built on LEO with a novel Situated Spatial Alignment Module that explicitly aligns 3D object representations with textual descriptions. Through extensive experiments on Spartun3D, SQA3D, and MS3D navigation, the approach yields consistent improvements in situated captioning, QA, and embodied navigation—demonstrating better spatial reasoning, reduced biases, and stronger generalization. The results validate both the dataset design and the alignment mechanism as effective means to enhance situated 3D understanding in LLMs, with implications for scalable training of embodied agents. Overall, Spartun3D offers a scalable path to endow 3D-LMMs with robust, context-sensitive spatial reasoning across tasks.

Abstract

Integrating the 3D world into large language models (3D-based LLMs) has been a promising research direction for 3D scene understanding. However, current 3D-based LLMs fall short in situated understanding due to two key limitations: 1) existing 3D datasets are constructed from a global perspective of the 3D scenes and lack situated context. 2) the architectures of existing 3D-based LLMs lack explicit alignment between the spatial representations of 3D scenes and natural language, limiting their performance in tasks requiring precise spatial reasoning. We address these issues by introducing a scalable situated 3D dataset, named Spartun3D, that incorporates various situated spatial reasoning tasks. Furthermore, we propose Spartun3D-LLM, built on an existing 3D-based LLM but integrated with a novel situated spatial alignment module, aiming to enhance the alignment between 3D visual representations and their corresponding textual descriptions. Experimental results demonstrate that both our proposed dataset and alignment module significantly enhance the situated spatial understanding of 3D-based LLMs.

Figures (13)

• Figure 1: Illustration of situated scene understanding of Spartun3D-LLM compared to other 3D-based LLMs.
• Figure 2: Examples of Spartun3D. Green box and arrow show the standing point and orientation.
• Figure 3: Illustration of Spartun3D Dataset Construction Process. Given a 3D scene: a) we first select a pivot object (e.g., sofa) and a referent object (e.g., table) to define the situation; b) we create situated scene graph based on situation, incorporating various spatial relationships (See Fig. \ref{['fig:spatial info']}); c) we use the scene graph to prompt GPT-4o (shown in gray box) to generate data; e) we utilize different prompting strategies for generating situated captions and QA pairs.
• Figure 4: Standing Point and Orientation Selection.
• Figure 5: Spatial information in Situated Scene Graph. The red dot and green arrow show the standing point and orientation, respectively. In this example, the pivot object is the "sofa", the referent object is the "TV", and the surrounding objects include the "table" and "cabinet".