Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting

TL;DR

This work introduces Segment then Splat, a 3D open-vocabulary segmentation framework that assigns object-specific Gaussians before reconstruction to ensure a consistent object–Gaussian mapping in both static and dynamic scenes. By coupling a robust object-tracking pipeline, per-object Gaussian initialization across three granularity levels, and an object-aware optimization with CLIP-based open-vocabulary embeddings, the approach achieves precise 3D segmentations and efficient one-pass reconstruction without requiring a separate language field. The method demonstrates state-of-the-art performance on static and dynamic datasets, with clear advantages in handling motion, reducing geometric/semantic ambiguities, and enabling efficient open-language querying. These results have practical implications for robotics and AR by enabling accurate, language-driven scene understanding in changing environments.

Abstract

Open-vocabulary querying in 3D space is crucial for enabling more intelligent perception in applications such as robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading to multi-view inconsistencies and poor 3D object retrieval. Moreover, they are limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling. In this paper, we propose Segment then Splat, a 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. Segment then Splat reverses the long established approach of "segmentation after reconstruction" by dividing Gaussians into distinct object sets before reconstruction. Once reconstruction is complete, the scene is naturally segmented into individual objects, achieving true 3D segmentation. This design eliminates both geometric and semantic ambiguities, as well as Gaussian-object misalignment issues in dynamic scenes. It also accelerates the optimization process, as it eliminates the need for learning a separate language field. After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments one various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.

Figures (8)

• Figure 1: Traditional 3D Open-Vocabulary Segmentation vs. our Segment-then-Splat Pipeline. (a) The traditional Splat-then-Segment pipeline learns a language field alongside the reconstruction of the entire scene. During object queries, it renders Gaussian language embeddings into a 2D feature map to identify relevant pixels based on the input text embedding. (b) In contrast, our Segment-then-Splat pipeline first initialize Gaussians into object-specific sets before reconstruction, ensuring a more precise object-Gaussian correspondence and improving segmentation accuracy.
• Figure 2: Demonstration of Segment then Splat pipeline. We first extracts multi-view masks for each object through a robust tracking module, then object IDs are assigned to each initial Gaussian based on these masks, forming distinct object-specific sets. During optimization, object specific loss $\mathcal{L}_{\text{obj}}$ is used to enforce Gaussian-object correspondence and thus resulting in more accurate object geometries. Finally, a CLIP embedding is assigned to each Gaussian group for open-vocabulary queries.
• Figure 3: A demonstration of how the optimization order affects reconstruction. Optimizing small-level objects first preserves both small- and middle-level structures, while starting with middle-level ones leads to well-maintained middle-level but chaotic small-level regions due to lack of supervision.
• Figure 4: Qualitative comparison on static scenes. Compared to baseline methods, our approach accurately retrieves the correct object and produces sharper segmentation boundaries. In contrast, 2D pixel-based methods exhibit ambiguous boundaries, while OpenGaussian either misses parts of the object or incorrectly groups irrelevant objects together.
• Figure 5: Comparison between 2D pixel-based segmentation and our 3D segmentation. Unlike 2D pixel-based methods, which are limited by occlusions, our approach can retrieve the complete object even from an occluded view.