Training-Free Hierarchical Scene Understanding for Gaussian Splatting with Superpoint Graphs

TL;DR

This work tackles open-vocabulary 3D scene understanding by eliminating iterative per-view optimization and enforcing 3D semantic consistency. It introduces a training-free pipeline that constructs a multi-level semantic field on a superpoint graph derived from Gaussian primitives, leveraging SAM-guided contrastive partitioning and rendering-guided reprojection to fuse 2D semantic cues into 3D. The approach delivers state-of-the-art open-vocabulary segmentation while achieving over $30\times$ faster semantic-field reconstruction than prior methods, confirmed by experiments on LERF-OVS, 3DOVS, and ScanNet. The hierarchical design enables both coarse and fine-grained language-driven perception and supports interactive, parts-based scene editing in 3D, with strong practical implications for AR and robotics.

Abstract

Bridging natural language and 3D geometry is a crucial step toward flexible, language-driven scene understanding. While recent advances in 3D Gaussian Splatting (3DGS) have enabled fast and high-quality scene reconstruction, research has also explored incorporating open-vocabulary understanding into 3DGS. However, most existing methods require iterative optimization over per-view 2D semantic feature maps, which not only results in inefficiencies but also leads to inconsistent 3D semantics across views. To address these limitations, we introduce a training-free framework that constructs a superpoint graph directly from Gaussian primitives. The superpoint graph partitions the scene into spatially compact and semantically coherent regions, forming view-consistent 3D entities and providing a structured foundation for open-vocabulary understanding. Based on the graph structure, we design an efficient reprojection strategy that lifts 2D semantic features onto the superpoints, avoiding costly multi-view iterative training. The resulting representation ensures strong 3D semantic coherence and naturally supports hierarchical understanding, enabling both coarse- and fine-grained open-vocabulary perception within a unified semantic field. Extensive experiments demonstrate that our method achieves state-of-the-art open-vocabulary segmentation performance, with semantic field reconstruction completed over $30\times$ faster. Our code will be available at https://github.com/Atrovast/THGS.

Figures (5)

• Figure 1: Framework Overview. a) Preprocessing: Scene reconstruction and extraction of 2D semantic maps. b) Contrastive Gaussian Partitioning: A Gaussian adjacency graph is created, and its edge weights are adjusted using SAM-guided contrastive cues. The scene is then partitioned into superpoints. c) Hierarchical Semantic Representation: Superpoints are progressively merged to form a multi-level superpoint graph, while semantic features are reprojected onto each level. d) Query and Decomposition: The resulting hierarchical graph enables open-vocabulary query and part-based decomposition of scene objects.
• Figure 2: Qualitative comparisons of open-vocabulary segmentation on multi-view 2D images. We compare our method with LEGaussians shi2023legs, LangSplat qin2023langsplat, and OpenGaussian wu2024opengaussian. Each scene includes an object- and a part-level query. Query results are highlighted, and corresponding ground-truth regions are marked with orange (object) and blue (part) bounding boxes.
• Figure 3: Qualitative comparison of open-vocabulary 3D segmentation. We compare our method with OpenGaussian wu2024opengaussian and LangSplat qin2023langsplat by visualizing the predicted 3D Gaussian primitives. The queried regions are annotated with colored bounding boxes on the original images to indicate object locations.
• Figure 4: Interactive Segmentation Results. Starting from a point prompt (red circle), our method enables coarse-to-fine object segmentation by traversing the hierarchical superpoint graph—retrieving regions from fine-grained components ($\mathcal{S}_1$) to complete objects ($\mathcal{S}_3$). This demonstrates the flexibility of our hierarchical representation for intuitive part-level interaction.
• Figure 5: Qualitative comparisons of open-vocabulary on 2D images (top four rows) and 3D Gaussian primitives (bottom three rows). We show results from our method alongside LEGaussians shi2023legs, LangSplat qin2023langsplat, GOI qu2024goi, and OpenGaussian wu2024opengaussian. Our approach delivers coherent 3D understanding and effectively supports open-vocabulary querying at both object and part levels. The queried foreground regions are highlighted, and the prompts are shown on the left side of each row.