Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding

Yan Wang, Baoxiong Jia, Ziyu Zhu, Siyuan Huang

TL;DR

This work tackles open-vocabulary 3D scene understanding by introducing MPEC, a Masked Point-Entity Contrastive framework that learns entity-aware 3D representations. It jointly optimizes cross-view point-to-entity contrast and entity-to-language alignment, using masked point augmentations and a VL-adapter to fuse 3D features with language. MPEC achieves state-of-the-art open-vocabulary segmentation on ScanNet and demonstrates strong zero-shot transfer and data-efficient fine-tuning across eight datasets and diverse downstream tasks, including grounding and QA. The results highlight the value of incorporating 3D spatial information and robust language alignment, suggesting a scalable path toward versatile 3D vision-language foundations for embodied AI.

Abstract

Open-vocabulary 3D scene understanding is pivotal for enhancing physical intelligence, as it enables embodied agents to interpret and interact dynamically within real-world environments. This paper introduces MPEC, a novel Masked Point-Entity Contrastive learning method for open-vocabulary 3D semantic segmentation that leverages both 3D entity-language alignment and point-entity consistency across different point cloud views to foster entity-specific feature representations. Our method improves semantic discrimination and enhances the differentiation of unique instances, achieving state-of-the-art results on ScanNet for open-vocabulary 3D semantic segmentation and demonstrating superior zero-shot scene understanding capabilities. Extensive fine-tuning experiments on 8 datasets, spanning from low-level perception to high-level reasoning tasks, showcase the potential of learned 3D features, driving consistent performance gains across varied 3D scene understanding tasks. Project website: https://mpec-3d.github.io/

Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding

TL;DR

This work tackles open-vocabulary 3D scene understanding by introducing MPEC, a Masked Point-Entity Contrastive framework that learns entity-aware 3D representations. It jointly optimizes cross-view point-to-entity contrast and entity-to-language alignment, using masked point augmentations and a VL-adapter to fuse 3D features with language. MPEC achieves state-of-the-art open-vocabulary segmentation on ScanNet and demonstrates strong zero-shot transfer and data-efficient fine-tuning across eight datasets and diverse downstream tasks, including grounding and QA. The results highlight the value of incorporating 3D spatial information and robust language alignment, suggesting a scalable path toward versatile 3D vision-language foundations for embodied AI.

Abstract

Open-vocabulary 3D scene understanding is pivotal for enhancing physical intelligence, as it enables embodied agents to interpret and interact dynamically within real-world environments. This paper introduces MPEC, a novel Masked Point-Entity Contrastive learning method for open-vocabulary 3D semantic segmentation that leverages both 3D entity-language alignment and point-entity consistency across different point cloud views to foster entity-specific feature representations. Our method improves semantic discrimination and enhances the differentiation of unique instances, achieving state-of-the-art results on ScanNet for open-vocabulary 3D semantic segmentation and demonstrating superior zero-shot scene understanding capabilities. Extensive fine-tuning experiments on 8 datasets, spanning from low-level perception to high-level reasoning tasks, showcase the potential of learned 3D features, driving consistent performance gains across varied 3D scene understanding tasks. Project website: https://mpec-3d.github.io/
Paper Structure (45 sections, 14 equations, 4 figures, 13 tables)

This paper contains 45 sections, 14 equations, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Open-Vocabulary 3D Scene Understanding
  4. Pre-training for 3D Point Clouds
  5. Method
  6. Point-to-Entity Alignment
  7. Cross-view Feature Extraction
  8. Point-to-Entity Contrastive Learning
  9. Entity-to-Language Alignment
  10. Point Features Merging
  11. Language Feature Extraction
  12. Entity-to-Language Contrastive Learning
  13. Model Learning
  14. Training Data
  15. Learning Objectives
  16. ...and 30 more sections

Figures (4)

  • Figure 1: The Overall Pipeline of MPEC. Given a 3D point cloud as input, we predict entity mask proposals and generate text descriptions for each entity. Then different views of the scene are randomly masked and we replace the masked point features with learnable embeddings. A 3D UNet then extracts per-point features. Guided by the entity masks, cross-view point-to-entity contrastive learning is conducted to enforce cross-view consistency and distinguishment across different entities. Then we merge the point features from two views and align the feature dimension of points with text features extracted by CLIP with a VL-Adapter. Finally, entity-to-language contrastive learning is performed for open-vocabulary 3D scene understanding.
  • Figure 2: Qualitative Results on ScanNetdai2017scannet. We show relatively good and bad cases in blue and red.
  • Figure A.3: More Qualitative Results on ScanNetdai2017scannet.
  • Figure A.4: More Qualitative Results on ScanNetdai2017scannet.