Split&Splat: Zero-Shot Panoptic Segmentation via Explicit Instance Modeling and 3D Gaussian Splatting

TL;DR

Split&Splat tackles the lack of object-level semantics in 3D Gaussian splatting by explicitly modeling each object first and then reconstructing the scene. It introduces a two-stage pipeline: Split generates view-consistent 2D instance masks with depth-driven propagation to obtain 3D-consistent labels, and Splat reconstructs each object independently via per-instance Gaussian splatting before merging with an instance-aware objective. The approach yields state-of-the-art instance segmentation on ScanNetv2 among Gaussian-based methods and competitive open-vocabulary segmentation on LERF, while enabling editing operations such as removal and recoloring. This object-centric decomposition enhances 3D scene understanding, retrieval, and editing, with practical impact on robust 3D scene reasoning and manipulation.

Abstract

3D Gaussian Splatting (GS) enables fast and high-quality scene reconstruction, but it lacks an object-consistent and semantically aware structure. We propose Split&Splat, a framework for panoptic scene reconstruction using 3DGS. Our approach explicitly models object instances. It first propagates instance masks across views using depth, thus producing view-consistent 2D masks. Each object is then reconstructed independently and merged back into the scene while refining its boundaries. Finally, instance-level semantic descriptors are embedded in the reconstructed objects, supporting various applications, including panoptic segmentation, object retrieval, and 3D editing. Unlike existing methods, Split&Splat tackles the problem by first segmenting the scene and then reconstructing each object individually. This design naturally supports downstream tasks and allows Split&Splat to achieve state-of-the-art performance on the ScanNetv2 segmentation benchmark.

Figures (9)

• Figure 1: During Split, multi-view images are processed to estimate depth and instance masks, which are propagated in 3D to produce refined, view-consistent segmentations. A, B, and C denote the output of Split, which serves as an input to Splat.
• Figure 2: During Splat, each instance is reconstructed independently using 3DGS, then merged into a global model enriched with per-instance descriptors. A, B, and C denote the output of Split, which serves as an input to Splat.
• Figure 3: Examples of mask refinement process. For "old camera" the refined mask is $M^{sam}$ while for "stuffed bear" $\tilde{M}$ present a lower IoU with $M^{gs}$. For "sake bottle" and "spatula", $\tilde{M}$ is not present and is discovered throughout the refinement process. Selected masks for each object are highlighted with green borders.
• Figure 4: Multi-resolution mask generation. Exploiting SAM2's multi-scale grid ping point generation, we produce several masks that are subsequently merged to obtain the final result.
• Figure 5: Labeled point cloud ($P_{labeled}$) produced by mask propagation stage, ensuring 3D consistency.