OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding

TL;DR

OpenGaussian tackles 3D point-level open vocabulary understanding within the 3D Gaussian Splatting framework, addressing the tendency of prior work to focus on 2D pixel-level parsing. It learnings 3D-consistent instance features by leveraging SAM masks with intra-mask $L_s$ and inter-mask $L_c$ losses, without cross-frame associations, and introduces a two-level coarse-to-fine codebook to discretize features, producing lossless, distinguishable 3D instance representations. An instance-level 2D-3D association method links 3D Gaussians to 2D CLIP features via a joint IoU and feature-distance score $S_{ij}$, enabling open-vocabulary CLIP-based understanding without depth-based occlusion testing. Extensive experiments show improved open-vocabulary object selection in 3D space, enhanced 3D point-cloud understanding, and effective click-based object selection, highlighting practical impact for interactive robotics and scene understanding. Limitations include fixed Gaussian geometry and empirically chosen codebook sizes, with future work aimed at joint optimization of geometry and semantics and handling dynamic scenes.

Abstract

This paper introduces OpenGaussian, a method based on 3D Gaussian Splatting (3DGS) capable of 3D point-level open vocabulary understanding. Our primary motivation stems from observing that existing 3DGS-based open vocabulary methods mainly focus on 2D pixel-level parsing. These methods struggle with 3D point-level tasks due to weak feature expressiveness and inaccurate 2D-3D feature associations. To ensure robust feature presentation and 3D point-level understanding, we first employ SAM masks without cross-frame associations to train instance features with 3D consistency. These features exhibit both intra-object consistency and inter-object distinction. Then, we propose a two-stage codebook to discretize these features from coarse to fine levels. At the coarse level, we consider the positional information of 3D points to achieve location-based clustering, which is then refined at the fine level. Finally, we introduce an instance-level 3D-2D feature association method that links 3D points to 2D masks, which are further associated with 2D CLIP features. Extensive experiments, including open vocabulary-based 3D object selection, 3D point cloud understanding, click-based 3D object selection, and ablation studies, demonstrate the effectiveness of our proposed method. The source code is available at our project page: https://3d-aigc.github.io/OpenGaussian

Figures (15)

• Figure 1: Illustration of two text query strategies. (a) demonstrates the process of rendering a feature map and computing its similarity with text features to obtain a 2D mask, which is then used to generate a corresponding rendered image. (b) demonstrates the direct similarity computation of 3D Gaussian language features with text features, selecting Gaussian points with high similarity, and rendering to obtain a rendered image corresponding to the text.
• Figure 2: (a) We use the view-independent SAM boolean mask to train 3D instance features with 3D consistency for 3DGS.(b) We propose a two-level codebook for discretizing instance features from coarse to fine. (c) An instance-level 3D-2D feature association method to associate 2D CLIP features with 3D points without training.
• Figure 3: (a) Reference image/mesh; (b) instance features learned from Sec. \ref{['sec:method_ins_feat']}; (c)-(d) Point features after discretization by coarse-level and fine-level codebook (Sec. \ref{['sec:method_codebook']}).
• Figure 4: We render 3D instance points to an arbitrary training view, and associate 3D points with 2D masks based on the principle of joint IoU and feature similarity, which have already been extracted with mask-level CLIP features, thereby indirectly associating 3D points with CLIP features.
• Figure 5: Open-vocabulary 3D object selection on the LERF dataset. OpenGaussian outperforms LangSplat and LEGaussians in accurately identifying the 3D objects corresponding to text queries.