DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation

TL;DR

DenseScan introduces an automated pipeline that leverages 2D multimodal LLMs to produce dense, multi-level annotations for a 3D dataset (ScanNet), including object-level captions, scene-level descriptions, and scenario-driven questions. It pairs these annotations with a new 3D segmentation task, scenario-driven segmentation, and a corresponding Dense3D framework that fuses 3D point clouds, 2D multi-view imagery, and text to achieve deeper reasoning and more accurate masks. The work demonstrates that long, context-rich descriptions enhance 3D scene understanding and QA-style capabilities, and provides both the DenseScan dataset and Dense3D model to spur progress in 3D multimodal learning. Overall, DenseScan aims to bridge the gap between 2D-rich linguistic annotation and 3D spatial reasoning, opening new avenues for robotics, AR, and related applications.

Abstract

3D understanding is a key capability for real-world AI assistance. High-quality data plays an important role in driving the development of the 3D understanding community. Current 3D scene understanding datasets often provide geometric and instance-level information, yet they lack the rich semantic annotations necessary for nuanced visual-language tasks.In this work, we introduce DenseScan, a novel dataset with detailed multi-level descriptions generated by an automated pipeline leveraging multi-view 2D images and multimodal large language models (MLLMs). Our approach enables dense captioning of scene elements, ensuring comprehensive object-level descriptions that capture context-sensitive details. Furthermore, we extend these annotations through scenario-based question generation, producing high-level queries that integrate object properties, spatial relationships, and scene context. By coupling geometric detail with semantic richness, DenseScan broadens the range of downstream tasks, from detailed visual-language navigation to interactive question answering. Experimental results demonstrate that our method significantly enhances object-level understanding and question-answering performance in 3D environments compared to traditional annotation pipelines. We release both the annotated dataset and our annotation pipeline to facilitate future research and applications in robotics, augmented reality, and beyond. Through DenseScan, we aim to catalyze new avenues in 3D scene understanding, allowing researchers and practitioners to tackle the complexities of real-world environments with richer, more contextually aware annotations.

Figures (4)

• Figure 1: Overview of DenseScan Data Generation Pipeline. Stage 1: crop the target object and generate object-level description; Stage 2: highlight target object in a single frame and generate frame-level description to capture spacial dependencies with the surroundings; Stage 3: use multiple frames with target object highlighted to compose scene-level descriptions; Stage 4: Raw scene-level description need to go through a LLM for consistency checking, and in-consistent description will be eliminated; Stage 5: Adopt MLLM annotator to generate scenario-driven questions and verified by LLM and human before release to benchmark.
• Figure 2: DenseScan Dataset. (a) Two objects sampled from ScanNetdai2017scannet with dense-referring expressions and scenario-driven questions. Objects shown are "office chair" and "pictures", both highlighted in the point cloud. (b) The distribution of description lengths, with dotted lines marking quartiles. The x-axis is scaled logarithmically (base-10) to handle the long-tail distribution.
• Figure 3: Dense3D Model Architecture. Given the 3D scene and language description, the model first reprocess 3D scene into multimodal 2D data, including RGB frames, depth map and camera poses. Depth map and camera poses composes the 3D positional embedding, along with the RGB frames and text descriptions to be send into the LLM. Output from LLM include special [SEG] token that is crucial to guide the Query Decoder for mask generation.
• Figure 4: Visual results for 3D Scenario-Driven Segmentation Task. Each visual output presents a textual scenario-driven question, along with ground truth point cloud in green regions and predicted point cloud in orange regions.