ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities

TL;DR

This work tackles the challenge of grounding in 3D scenes under implicit user instructions by introducing 3D reasoning grounding and the ScanReason benchmark, which comprises over 10K question-answer-location triplets across five reasoning types. It presents ReGround3D, a two-module architecture where a visual-centric reasoning component powered by a Multi-modal Large Language Model informs a geometry-aware 3D grounding module, connected via a Chain-of-Grounding mechanism for iterative reasoning and grounding. The approach achieves state-of-the-art performance on traditional 3D visual grounding and demonstrates substantial gains in 3D reasoning grounding, especially when employing the CoG mechanism and instruction-tuning data. By enabling more natural and robust interpretation of implicit human instructions in 3D, this work advances embodied agents’ capabilities in open 3D environments and lays groundwork for richer human–machine interactions.

Abstract

Although great progress has been made in 3D visual grounding, current models still rely on explicit textual descriptions for grounding and lack the ability to reason human intentions from implicit instructions. We propose a new task called 3D reasoning grounding and introduce a new benchmark ScanReason which provides over 10K question-answer-location pairs from five reasoning types that require the synerization of reasoning and grounding. We further design our approach, ReGround3D, composed of the visual-centric reasoning module empowered by Multi-modal Large Language Model (MLLM) and the 3D grounding module to obtain accurate object locations by looking back to the enhanced geometry and fine-grained details from the 3D scenes. A chain-of-grounding mechanism is proposed to further boost the performance with interleaved reasoning and grounding steps during inference. Extensive experiments on the proposed benchmark validate the effectiveness of our proposed approach.

Figures (5)

• Figure 1: For an embodied agent, they not only need to be able to understand the 3D environment and complex human instructions but also localize the target objects for interaction and navigation. Although GPT-4 (GPT-4V) have strong text (multi-modal) reasoning abilities, they lack the ability to directly perceive the 3D scene, understand the 3D spatial relationships, and output corresponding target object locations. Instead, our proposed method ReGround3D has the 3D perception, reasoning, and grounding capabilities in the real-world 3D environment.
• Figure 2: The left side figure shows the overall of our ScanReason dataset. For each reasoning category, we designed different prompts to generate corresponding questions. And the right side figure shows the differences between the traditional 3D visual grounding task and our proposed 3D reasoning grounding task.
• Figure 3: The pipeline of ReGround3D. Given the 3D scene and human instruction, the visual reasoning module first performs joint 3D scene and instruction reasoning, and then guide the 3D grounding module to look-back the 3D scene and perform target object location.
• Figure 4: Illustration of Chain-of-Grounding (CoG) Mechanism
• Figure 5: Visualization comparison of 3D reasoning grounding capability between ReGround3D and 3D-LLM. Our method could achieve much more accurate grounding results which satisfy the implicit question intention, and give the corresponding explanation at the same time. More illustrations are given in the Appendix.