Learning Segmented 3D Gaussians via Efficient Feature Unprojection for Zero-shot Neural Scene Segmentation

TL;DR

The paper tackles zero-shot neural scene segmentation with 3D Gaussian representations, addressing cross-view inconsistencies and overparameterization in prior methods. It introduces CoSegGaussians, featuring a shared segmentation decoder and a Feature Unprojection & Fusion pipeline that lifts image-based semantic features from 2D foundation models into the 3D Gaussian field. A pixel-wise CoSeg Loss combined with 3D regularization enhances robustness to noisy cross-view labels, yielding about a 10% mIoU gain over baselines and substantially faster inference than NeRF-based approaches. The method leverages DINO and LSeg features and demonstrates improved segmentation quality, efficiency, and potential for annotation-free 3D scene understanding and manipulation.

Abstract

Zero-shot neural scene segmentation, which reconstructs 3D neural segmentation field without manual annotations, serves as an effective way for scene understanding. However, existing models, especially the efficient 3D Gaussian-based methods, struggle to produce compact segmentation results. This issue stems primarily from their redundant learnable attributes assigned on individual Gaussians, leading to a lack of robustness against the 3D-inconsistencies in zero-shot generated raw labels. To address this problem, our work, named Compact Segmented 3D Gaussians (CoSegGaussians), proposes the Feature Unprojection and Fusion module as the segmentation field, which utilizes a shallow decoder generalizable for all Gaussians based on high-level features. Specifically, leveraging the learned Gaussian geometric parameters, semantic-aware image-based features are introduced into the scene via our unprojection technique. The lifted features, together with spatial information, are fed into the multi-scale aggregation decoder to generate segmentation identities for all Gaussians. Furthermore, we design CoSeg Loss to boost model robustness against 3D-inconsistent noises. Experimental results show that our model surpasses baselines on zero-shot semantic segmentation task, improving by ~10% mIoU over the best baseline. Code and more results will be available at https://David-Dou.github.io/CoSegGaussians.

Figures (9)

• Figure 1: Comparison of performance and inference speed for zero-shot scene segmentation.
• Figure 2: Failure for compact 3D segmentation caused by cross-view inconsistency (selected regions) in previous Gaussian-based method.
• Figure 3: Overview of our method. Training process takes a two-stage strategy. In Gaussian Building Stage, geometric ($\boldsymbol{x, \Sigma, \alpha}$) and appearance ($\boldsymbol{c}$) attributes are optimized following 3DGS kerbl20233d. Next in Segmentation Learning Stage, segmentation identities $\boldsymbol{s}$ are learned with the Feature Unprojection & Fusion module. It takes the extracted 3D spatial information as well as the image-based semantic-aware features as input, as the latter is introduced into 3D Gaussians efficiently via our unprojection technique. After fusing the above features with the multi-scale aggregation network, $\boldsymbol{s}$ for all Gaussians are generated, which are supervised by comparing the rendered segmentation map with the zero-shot segmented and further associated raw label, guided by our CoSeg Loss. For inference, with Gaussian attributes $\boldsymbol{x, \Sigma, \alpha}$ and $\boldsymbol{s}$ learned, segmentation maps are rendered on novel views.
• Figure 4: Feature unprojection. 3D Gaussian $i$'s feature can be acquired by gathering 2D features on pixels from training viewpoints if the pixel is affected by $i$.
• Figure 5: Multi-scale feature fusion for segmentation identity generation. (LFA: Local Feature Aggregation, RS & US: Random & Up Sampling, FC: Fully Connected Layer)