Empowering Large Language Models with 3D Situation Awareness

TL;DR

This work tackles the lack of egocentric, situation-aware grounding in 3D LLMs by introducing View2Cap, a scalable dataset generated from RGB-D trajectories using 2D Vision-Language Models to produce captions and QA that encode an observer's viewpoint. A Situation Grounding (SG) module is proposed to explicitly predict the observer's position $\mathbf{s}^{pos} \in \mathbb{R}^3$ and rotation $\mathbf{s}^{rot} \in \mathbb{R}^4$ by leveraging object anchors and rotation-bin classification, converting pose estimation into tractable offset and discretized-angle tasks. Through a three-stage training regime (feature alignment, grounding supervision, and downstream instruction-tuning with LoRA on the base LLM $\text{LLaMa 3.1}$), the approach achieves significant gains across 3D VL tasks such as captioning, grounding, and QA on datasets like Scan2Cap, ScanQA, and SQA3D. The View2Cap dataset, combined with SG, expands 3D situational understanding capabilities while reducing manual labeling burden, with practical impact for embodied AI in real-world navigation and interaction scenarios.

Abstract

Driven by the great success of Large Language Models (LLMs) in the 2D image domain, their applications in 3D scene understanding has emerged as a new trend. A key difference between 3D and 2D is that the situation of an egocentric observer in 3D scenes can change, resulting in different descriptions (e.g., ''left" or ''right"). However, current LLM-based methods overlook the egocentric perspective and simply use datasets from a global viewpoint. To address this issue, we propose a novel approach to automatically generate a situation-aware dataset by leveraging the scanning trajectory during data collection and utilizing Vision-Language Models (VLMs) to produce high-quality captions and question-answer pairs. Furthermore, we introduce a situation grounding module to explicitly predict the position and orientation of observer's viewpoint, thereby enabling LLMs to ground situation description in 3D scenes. We evaluate our approach on several benchmarks, demonstrating that our method effectively enhances the 3D situational awareness of LLMs while significantly expanding existing datasets and reducing manual effort.

Figures (10)

• Figure 1: Illustration of 3D Situation Awareness. The LLMs can accurately ground situation descriptions to the observer's position and orientation, enabling context-aware question answering based on the observer's viewpoint.
• Figure 2: Overview of our method. The left part illustrates the process of region-text alignment. Paired point cloud and caption data are generated using VLM and RGB-D videos. The LLM is fine-tuned to align features from the point cloud encoder and generated region caption. For situation grounding, the region caption is fed into the LLM to predict the viewpoint of the observer $\mathbf{s}^{pos}$ and $\mathbf{s}^{rot}$. The right part shows the situation-aware instruction tuning process, where QA data is generated using multi-view images and corresponding actions.
• Figure 3: Situation prediciton. We treat each object as anchor (blue), where the center is $\mathbf{a}_k^{\text{pos}}$. We set each $\mathbf{a}_k^{\text{rot}}$ point to the center of the room. Ground truth position and rotation are shown in green. The dotted line shows the offset from $\mathbf{a}_k^{\text{pos}}$ to $\mathbf{s}_k^{\text{pos}}$. The solid arrow shown the predicted rotation is rotated by $\theta$ from $\mathbf{a}_k^{\text{rot}}$.
• Figure 4: Examples of our View2Cap situation captions against SceneVerse. We mark facts in RGB]153, 255, 153green and spatial relations in RGB]204, 236, 255blue.
• Figure 5: Dataset statistics.