Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

Xujing Tao, Chuxin Wang, Yubo Ai, Zhixin Cheng, Zhuoyuan Li, Liangsheng Liu, Yujia Chen, Xinjun Li, Qiao Li, Wenfei Yang, Tianzhu Zhang

Abstract

Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide, a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide.

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

Abstract

Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide, a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide.

Paper Structure

This paper contains 19 sections, 10 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Closed-Set 3D Scene Understanding
  4. Large-Scale Pretrained Models
  5. Open-Vocabulary 3D Semantic Segmentation
  6. Method
  7. Framework Overview
  8. Feature Extraction and 2D-3D Alignment
  9. Hierarchical Geometry-Guided Modules
  10. Uncertainty-Based Superpoint Distillation
  11. Instance-Level Mask Reconstruction
  12. Inter-Instance Relation Consistency
  13. Model Training and Inference
  14. Experiments
  15. Experimental Setup
  16. ...and 4 more sections

Figures (5)

  • Figure 1: (a) Occlusions and ambiguous boundaries in 2D predictions lead to inaccurate 3D segmentation, whereas our geometry-guided modeling effectively rectifies these errors. (b) The baseline suffers from the lack of geometric relation modeling, whereas our method exploits inter-instance geometric constraints to produce accurate predictions.
  • Figure 2: Overview of our proposed GeoGuide framework. GeoGuide integrates hierarchical geometric priors to achieve geometry–semantic consistent open-vocabulary 3D segmentation
  • Figure 3: Preliminary Experiments. Comparison between OpenScene and our preliminary attempt under different 2D models. LS: LSeg, OS: OpenSeg.
  • Figure 4: Overview of the USD. Uncertainty prediction from 3D geometry and 2D semantics guides the distillation of semantic knowledge.
  • Figure 5: Visualization of semantic segmentation results on ScanNet v2 and Matterport3D.