SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition

TL;DR

The paper tackles the problem of rough and incomplete object boundaries when segmenting 3D Gaussian Splatting representations. It introduces SAGD, a training-free boundary-enhanced segmentation pipeline that leverages Gaussian Decomposition to handle boundary Gaussians and lifts a 2D SAM-based segmentation into 3D via multi-view prompts and a simple label voting scheme. The approach achieves high-quality 3D segmentation with significantly reduced computation compared to training-based methods and demonstrates applicability to scene editing and collision detection. Comprehensive experiments across SPIn-NeRF, LLFF, mip-NeRF360, and 3D-GS scenes validate improved boundary handling and robust performance, along with thorough ablations on decomposition, view counts, and hyperparameters.

Abstract

3D Gaussian Splatting has emerged as an alternative 3D representation for novel view synthesis, benefiting from its high-quality rendering results and real-time rendering speed. However, the 3D Gaussians learned by 3D-GS have ambiguous structures without any geometry constraints. This inherent issue in 3D-GS leads to a rough boundary when segmenting individual objects. To remedy these problems, we propose SAGD, a conceptually simple yet effective boundary-enhanced segmentation pipeline for 3D-GS to improve segmentation accuracy while preserving segmentation speed. Specifically, we introduce a Gaussian Decomposition scheme, which ingeniously utilizes the special structure of 3D Gaussian, finds out, and then decomposes the boundary Gaussians. Moreover, to achieve fast interactive 3D segmentation, we introduce a novel training-free pipeline by lifting a 2D foundation model to 3D-GS. Extensive experiments demonstrate that our approach achieves high-quality 3D segmentation without rough boundary issues, which can be easily applied to other scene editing tasks.

Figures (9)

• Figure 1: (a)The training of 3D-GS doesn't consider the structure of objects, leading to the ambiguous geometry; (b)Direct segmentation without Gaussian Decomposition processing will result in rough boundary segmentation; (3) The recent SAGA also has incomplete segmentation caused by the same issue; (d) Our full pipeline considers this issue and can achieve better segmentation.
• Figure 2: Pipeline of our proposed method. (a) Given a set of clicked points on the $1^{st}$ reference view, we utilize SAM to generate masks for corresponding objects under every view automatically; (b) For every view, Gaussian Decomposition is first performed to address the issue of boundary roughness and then label propagation is implemented to assign binary labels to each 3D Gaussian; (c) Finally, with assigned 3D labels from all views, we adopt a simple yet effective voting strategy to determine the segmented Gaussians.
• Figure 3: Illustration of the Gaussian Decomposition process. It involves two basic steps: first, to find out the boundary Gaussians and then decompose these Gaussians.
• Figure 4: Qualitative results compared with SA3D SA3D and SAGA SAGA in different scenes (LERF-figurines lerf, SPIn-NeRF-Orchids spin-nerf, LERF-dozer-nerfgun-waldo lerf). We enlarge the boxed area on the right for a better visualization.
• Figure 5: 3D segmentation with text prompts in Mip-NeRF360-garden mip-nerf.