PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understanding

TL;DR

PanoGS addresses the challenge of 3D panoptic open vocabulary scene understanding by learning a smooth, 3D language feature field through a latent pyramid tri-plane and a 3D language decoder, avoiding 2D-to-3D feature biases. It introduces language-guided graph cuts to form geometrically and semantically coherent super-primitives, and uses multi-view SAM-based affinity for robust, 3D-consistent instance segmentation via progressive clustering. The method yields improved semantic and panoptic metrics on Replica and ScanNetV2, demonstrating strong open-vocabulary capabilities and robust instance discrimination in indoor scenes. This approach advances 3D scene understanding by enabling true 3D instance-level open vocabulary reasoning with scalable, view-consistent representations.

Abstract

Recently, 3D Gaussian Splatting (3DGS) has shown encouraging performance for open vocabulary scene understanding tasks. However, previous methods cannot distinguish 3D instance-level information, which usually predicts a heatmap between the scene feature and text query. In this paper, we propose PanoGS, a novel and effective 3D panoptic open vocabulary scene understanding approach. Technically, to learn accurate 3D language features that can scale to large indoor scenarios, we adopt the pyramid tri-plane to model the latent continuous parametric feature space and use a 3D feature decoder to regress the multi-view fused 2D feature cloud. Besides, we propose language-guided graph cuts that synergistically leverage reconstructed geometry and learned language cues to group 3D Gaussian primitives into a set of super-primitives. To obtain 3D consistent instance, we perform graph clustering based segmentation with SAM-guided edge affinity computation between different super-primitives. Extensive experiments on widely used datasets show better or more competitive performance on 3D panoptic open vocabulary scene understanding. Project page: \href{https://zju3dv.github.io/panogs}{https://zju3dv.github.io/panogs}.

Figures (8)

• Figure 1: Open Vocabulary 3D Panoptic Scene Understanding. Visualization of open-vocabulary semantic segmentation (yellow boxes) and object query with text toilet (red boxes). Our PanoGS can achieve more accurate segmentation results and generate 3D instance-level results for open-vocabulary text queries, unlike previous methods that generate heatmaps between scene features and text queries.
• Figure 2: Overview of our approach. (a) Given posed RGB-D images, we reconstruct the scene with 3D Gaussian primitives, and each primitive is associated with additional latent language code $g$ generated from a latent continuous pyramid tri-plane feature space. (b) After the geometry reconstruction, we obtain 2D fused primitive-level features and confidences via back projection, which is used for efficient 3D language feature regression and latent pyramid tri-plane and 3D decoder optimization. (c) We perform a language-guided graph cuts algorithm to construct super-primitive and use the 2D instance mask generated by SAM sam to conduct progressive graph clustering.
• Figure 3: By projecting primitives inside different graph vertices into $k$-th 2D SAM mask, $\mathcal{V}_1$ and $\mathcal{V}_2$ have similar mask label distributions, while $\mathcal{V}_3$ has different mask label distributions. So, we can cluster $\mathcal{V}_1$ and $\mathcal{V}_2$ into the same category based on the distance between the distributions.
• Figure 4: Qualitative 3D semantic segmentation comparison of ScanNetV2 dai:2017:scannet. Our approach outperforms recent 3DGS-based approaches, LangSplat qin2024langsplat and OpenGaussian wu2024opengaussian, by a large margin. Compared with OpenScene Peng2023OpenScene, we can achieve better segmentation results on thing-level objects.
• Figure 5: Qualitative 3D panoptic segmentation comparison. We show two reconstructed panoptic maps selected from ScanNetV2 dai:2017:scannet and Replica julian:2019:replica datasets.