Language-Grounded Indoor 3D Semantic Segmentation in the Wild

TL;DR

The paper addresses the gap between real-world scene diversity and existing 3D semantic segmentation benchmarks by introducing ScanNet200, a 200-class indoor dataset. It proposes language-grounded 3D feature learning that anchors geometric features to CLIP text embeddings via a cross-modal contrastive objective, enabling robust representations across many classes. To tackle natural class imbalance and limited annotations, the approach incorporates instance-based data balancing and a class-balanced loss during fine-tuning. Empirically, the method yields substantial gains over 3D pre-training baselines, including strong improvements in limited-data regimes and in 3D instance segmentation, demonstrating the practicality of language-grounded 3D perception for wild indoor environments.

Abstract

Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on available datasets. However, current 3D semantic segmentation benchmarks contain only a small number of categories -- less than 30 for ScanNet and SemanticKITTI, for instance, which are not enough to reflect the diversity of real environments (e.g., semantic image understanding covers hundreds to thousands of classes). Thus, we propose to study a larger vocabulary for 3D semantic segmentation with a new extended benchmark on ScanNet data with 200 class categories, an order of magnitude more than previously studied. This large number of class categories also induces a large natural class imbalance, both of which are challenging for existing 3D semantic segmentation methods. To learn more robust 3D features in this context, we propose a language-driven pre-training method to encourage learned 3D features that might have limited training examples to lie close to their pre-trained text embeddings. Extensive experiments show that our approach consistently outperforms state-of-the-art 3D pre-training for 3D semantic segmentation on our proposed benchmark (+9% relative mIoU), including limited-data scenarios with +25% relative mIoU using only 5% annotations.

Figures (8)

• Figure 1: We present the ScanNet200 benchmark, which studies 200-class 3D semantic segmentation -- an order of magnitude more categories than previous 3D scene understanding benchmarks. To address this challenging task, we propose to guide 3D feature learning by anchoring it to the richly-structured text embedding space of CLIP for the semantic class labels. This results in improved 3D semantic segmentation across the large set of class categories.
• Figure 2: During pre-training, we guide 3D feature learning by mapping learned features to text encoded anchors of the corresponding semantic labels, constructed by a constrastive loss between text and 3D. This establishes a more robust 3D feature representation space guided by the rich structure of the text embeddings.
• Figure 3: Our instance sampling augments scenes during training with by placing rarely-seen class category instances into them, breaking unduly specific context dependencies that can be easily learned from only a few examples.
• Figure 4: Class category distribution for our ScanNet200 Benchmark showing number of instances per category; note that the frequencies are given on log-scale and ordered by number of instances per category.
• Figure 5: 3D semantic segmentation under varying amounts of limited annotations. Even when considering only a small number of annotated surface points for our supervised language-guided 3D pre-training, our approach improves notably over the state-of-the-art 3D pre-training of CSC scene_contrast.