SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes

TL;DR

The paper tackles the challenge of grounded reasoning in 3D scenes by introducing SceneCOT, a four-stage framework that decomposes complex questions into task recognition, region localization, entity grounding, and grounded reasoning. It pairs SceneCOT with SceneCOT-185K, a large-scale dataset of 185K stepwise reasoning traces linking language to 3D context, enabling supervised learning of interpretable reasoning traces. Through extensive experiments on MSQA and Beacon3D, SceneCOT achieves strong 3D QA performance and, crucially, substantially improves grounding–QA coherence by explicitly grounding intermediate steps before answering. The work contributes a practical methodology and dataset that advance grounded multimodal LLMs for 3D scene understanding and sets the stage for broader embodied AI applications.

Abstract

Existing research on 3D Large Language Models (LLMs) still struggles to achieve grounded question-answering, primarily due to the under-exploration of the mechanism of human-like scene-object grounded reasoning. This paper bridges the gap by presenting a novel framework. We first introduce a grounded Chain-of-Thought reasoning method in 3D scenes (SCENECOT), decoupling a complex reasoning task into simpler and manageable problems, and building corresponding visual clues based on multimodal expert modules. To enable such a method, we develop SCENECOT-185K, the first large-scale grounded CoT reasoning dataset, consisting of 185K high-quality instances. Extensive experiments across various complex 3D scene reasoning benchmarks demonstrate that our new framework achieves strong performance with high grounding-QA coherence. To the best of our knowledge, this is the first successful application of CoT reasoning to 3D scene understanding, enabling step-by-step human-like reasoning and showing potential for extension to broader 3D scene understanding scenarios.

Figures (16)

• Figure 1: Reasoning chain visualization of SceneCOT. Example of how SceneCOT decomposes a 3D question into step-by-step reasoning: from identifying the task type, localizing relevant objects, and grounding the target entity, to retrieving the visual clue and generating the final answer.
• Figure 2: SceneCOT framework. The model decomposes 3D scene reasoning into four steps: task recognition, spatial region recognition, entity grounding, and grounded reasoning. Each stage introduces explicit grounding signals (e.g., objects, attributes, spatial positions), ensuring step-by-step reasoning and improved grounding–QA coherence.
• Figure 3: Illustration of reasoning trace generation. An example of a navigation question and its corresponding reasoning trace. The process involves identifying the question type, grounding relevant objects, computing relative positions based on agent location and orientation, and generating the final answer through step-by-step reasoning.
• Figure 4: Ablation study. We ablate three key factors: (1) question type recognition, (2) region recognition, and (3) grounding loss. Results show that removing any of these components degrades performance, highlighting their importance for robust step-by-step reasoning.
• Figure 5: Visualization of qualitative examples of SceneCOT. We select two indicating sub-tasks Counting and Navigation to illustrate the reasoning traces of SceneCOT. Left:SceneCOT correctly constructs the visual clue and reasons the correct answer. Middle: SceneCOT correctly answers the question based on the accurate relative location. Right: Even though the visual clue exactly matches the correct entity, the model summarizes to the wrong answer owing to the limited reasoning capability.