Reasoning Matters for 3D Visual Grounding

TL;DR

This work tackles 3D visual grounding by removing the heavy annotation burden and enhancing reasoning capabilities. It introduces a fully automatic data pipeline that fabricates 3D scenes with structured, multi-step reasoning targets and uses GPT-4o for supervision data, followed by fine-tuning an open-source LLM (Reason3DVG-8B) to perform grounding with explicit reasoning. The approach achieves state-of-the-art-like performance on ScanRefer and NR3D using only a small fraction of data compared to prior methods, illustrating the critical role of reasoning supervision over sheer data scale. By decoupling data generation from manual annotation and demonstrating robust generalization, the work highlights the practical potential of reasoning-aware LLMs for open, cost-efficient 3D understanding.

Abstract

The recent development of Large Language Models (LLMs) with strong reasoning ability has driven research in various domains such as mathematics, coding, and scientific discovery. Meanwhile, 3D visual grounding, as a fundamental task in 3D understanding, still remains challenging due to the limited reasoning ability of recent 3D visual grounding models. Most of the current methods incorporate a text encoder and visual feature encoder to generate cross-modal fuse features and predict the referring object. These models often require supervised training on extensive 3D annotation data. On the other hand, recent research also focus on scaling synthetic data to train stronger 3D visual grounding LLM, however, the performance gain remains limited and non-proportional to the data collection cost. In this work, we propose a 3D visual grounding data pipeline, which is capable of automatically synthesizing 3D visual grounding data along with corresponding reasoning process. Additionally, we leverage the generated data for LLM fine-tuning and introduce Reason3DVG-8B, a strong 3D visual grounding LLM that outperforms previous LLM-based method 3D-GRAND using only 1.6% of their training data, demonstrating the effectiveness of our data and the importance of reasoning in 3D visual grounding.

Figures (8)

• Figure 1: Our method outperforms 3D-GRAND on the ScanRefer benchmark when using only 1.6% amount of their training data scale. Compared with 3D-GRAND, our proposed data pipeline features lower data collection cost, incorporate extra reasoning supervision for LLM, and achieve 25% better grounding accuracy.
• Figure 2: We propose a fully automatic data pipeline that can generate visual grounding queries and reasoning responses. The collected data are used to conduct LLM fine-tuning, which results in Reason3DVG-8B, a powerful LLM with strong 3D visual grounding ability.
• Figure 3: An illustration of our 3D scene layout generation pipeline, which includes 5 steps: 1) Set up an empty 3D scene with certain dimension. 2) Choose a spatial relationship and decide the anchor and distractor objects as well as their dimension and location. 3) Place them in the 3D scene. 4) Determine the target object from the distractors and query. 5) Generate more distractor objects to enrich 3D scene.
• Figure 4: Qualitative results from NR3D. Green boxes and red boxes indicate predictions from our Reason3DVG and the base model, respectively. All predicted bounding boxes are re-plotted for better clarity.
• Figure 5: The spatial relationship templates used in our data collection framework.