DC-Scene: Data-Centric Learning for 3D Scene Understanding

TL;DR

DC-Scene tackles the efficiency and data-scarcity hurdles in 3D scene understanding by introducing a CLIP-driven data quality filter and a dual-indicator quality mechanism coupled with a progressive curriculum. By selecting high-quality, semantically aligned scene-caption pairs and gradually incorporating more challenging samples, it achieves state-of-the-art CIDEr scores on ScanRefer and Nr3D while requiring roughly one-third of the training epochs. The approach is validated on multiple backbones (e.g., 3D CoCa, Vote2Cap-DETR++) and across two prominent datasets, demonstrating substantial training-time reductions without performance loss. This data-centric paradigm offers a generalizable path to more efficient 3D vision-language learning, with practical implications for robotics, AR/VR, and autonomous systems.

Abstract

3D scene understanding plays a fundamental role in vision applications such as robotics, autonomous driving, and augmented reality. However, advancing learning-based 3D scene understanding remains challenging due to two key limitations: (1) the large scale and complexity of 3D scenes lead to higher computational costs and slower training compared to 2D counterparts; and (2) high-quality annotated 3D datasets are significantly scarcer than those available for 2D vision. These challenges underscore the need for more efficient learning paradigms. In this work, we propose DC-Scene, a data-centric framework tailored for 3D scene understanding, which emphasizes enhancing data quality and training efficiency. Specifically, we introduce a CLIP-driven dual-indicator quality (DIQ) filter, combining vision-language alignment scores with caption-loss perplexity, along with a curriculum scheduler that progressively expands the training pool from the top 25% to 75% of scene-caption pairs. This strategy filters out noisy samples and significantly reduces dependence on large-scale labeled 3D data. Extensive experiments on ScanRefer and Nr3D demonstrate that DC-Scene achieves state-of-the-art performance (86.1 CIDEr with the top-75% subset vs. 85.4 with the full dataset) while reducing training cost by approximately two-thirds, confirming that a compact set of high-quality samples can outperform exhaustive training. Code will be available at https://github.com/AIGeeksGroup/DC-Scene.

Figures (5)

• Figure 1: Effect of CLIP-score–based data filtering on training cost. Filtering out the lowest-quality 80% of samples cuts training time by 75%, motivating our data-centric learning strategy.
• Figure 2: Framework of DC-Scene. Raw point cloud scene and candidate captions are first encoded by a Scene Encoder and a Text Encoder. Quality Scoring module computes the CLIP score for each scene–text pair. Dual-Indicator Quality (DIQ) Filter then selects samples that lie within a predefined quality region, retaining only the top-$k$ candidates per scene. These filtered representations are passed to the Curriculum Scheduler, which gradually feeds them into the Multi-Modal Decoder for caption generation. A feedback loop returns the caption loss and updated CLIP scores to refine the quality map, thereby closing the data-centric learning cycle.
• Figure 3: Ablation study comparing different Dual-Indicator Quality (DIQ) regions on the ScanRefer and Nr3D datasets.
• Figure 4: Data quality buckets visualized in 3-D embedding space.
• Figure 5: Qualitative comparison of scene captions. For three validation scenes from the ScanRefer chen2020scanrefer dataset, we present the rendered point cloud mesh (top row), followed by captions generated by three sources: the full-data baseline model (in pink), our DC-Scene model trained on the top-75% DIQ samples (in red), and the human-annotated ground truth (in green).