SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians

TL;DR

The introduction of SuperGSeg, a novel approach that fosters cohesive, context-aware scene representation by disentangling segmentation and language field distillation and outperforms prior works on both open-vocabulary object localization and semantic segmentation tasks.

Abstract

3D Gaussian Splatting has recently gained traction for its efficient training and real-time rendering. While the vanilla Gaussian Splatting representation is mainly designed for view synthesis, more recent works investigated how to extend it with scene understanding and language features. However, existing methods lack a detailed comprehension of scenes, limiting their ability to segment and interpret complex structures. To this end, We introduce SuperGSeg, a novel approach that fosters cohesive, context-aware scene representation by disentangling segmentation and language field distillation. SuperGSeg first employs neural Gaussians to learn instance and hierarchical segmentation features from multi-view images with the aid of off-the-shelf 2D masks. These features are then leveraged to create a sparse set of what we call Super-Gaussians. Super-Gaussians facilitate the distillation of 2D language features into 3D space. Through Super-Gaussians, our method enables high-dimensional language feature rendering without extreme increases in GPU memory. Extensive experiments demonstrate that SuperGSeg outperforms prior works on both open-vocabulary object localization and semantic segmentation tasks.

Figures (10)

• Figure 1: We present SuperGSeg, a novel method that clusters similar Gaussians into superpoint-like representations, termed supergs (supergs). SuperGSeg enables efficient integration of diverse feature fields for comprehensive 3D scene understanding. Left: Querying supergs' language features enables open-vocabulary 3D object selection, producing consistent 3D masks that extend beyond 2D visible surfaces, e.g., the leg of the sheep under the table. Middle: Grouping supergs by instance features enables promptable instance segmentation. Right: Further splitting instances via hierarchical features enables fine-grained hierarchical segmentation.
• Figure 2: SuperGSeg Overview. We initialize the 3D Gaussians from a sparse set of anchor points, each generating $k$ Gaussians with corresponding attributes. First, we train the appearance and segmentation features using RGB images and segmentation masks generated by SAM kirillov2023sam. Next, we use the segmentation features and their spatial positions to produce a sparse set of supergs, each carrying a 512-dimensional language feature. Finally, we train this high-dimensional language feature using a 2D feature map from CLIP radford2021clip.
• Figure 3: The architecture of the superg Cluster Network.
• Figure 4: Qualitative comparison of semantic segmentation predictions on the ScanNet v2 dataset dai2017scannet.
• Figure 5: Qualitative comparison on the LERF-OVS dataset kerr2023lerf for the open-vocabulary 3D object selection task. Text queries for each scene are displayed in quotation marks. SuperGSeg delivers more precise and less noisy segmentation masks.