OGScene3D: Incremental Open-Vocabulary 3D Gaussian Scene Graph Mapping for Scene Understanding

Abstract

Open-vocabulary scene understanding is crucial for robotic applications, enabling robots to comprehend complex 3D environmental contexts and supporting various downstream tasks such as navigation and manipulation. However, existing methods require pre-built complete 3D semantic maps to construct scene graphs for scene understanding, which limits their applicability in robotic scenarios where environments are explored incrementally. To address this challenge, we propose OGScene3D, an open-vocabulary scene understanding system that achieves accurate 3D semantic mapping and scene graph construction incrementally. Our system employs a confidence-based Gaussian semantic representation that jointly models semantic predictions and their reliability, enabling robust scene modeling. Building on this representation, we introduce a hierarchical 3D semantic optimization strategy that achieves semantic consistency through local correspondence establishment and global refinement, thereby constructing globally consistent semantic maps. Moreover, we design a long-term global optimization method that leverages temporal memory of historical observations to enhance semantic predictions. By integrating 2D-3D semantic consistency with Gaussian rendering contribution, this method continuously refines the semantic understanding of the entire scene. Furthermore, we develop a progressive graph construction approach that dynamically creates and updates both nodes and semantic relationships, allowing continuous updating of the 3D scene graphs. Extensive experiments on widely used datasets and real-world scenes demonstrate the effectiveness of our OGScene3D on open-vocabulary scene understanding.

Figures (10)

• Figure 1: Our OGScene3D achieves incremental open-vocabulary scene understanding by continuously updating scene graphs and semantic Gaussian maps.
• Figure 2: An overview of OGScene3D. Given sequential RGB-D images, our method achieves open-vocabulary semantic mapping through hierarchical 3D semantic optimization and long-term global optimization. At the same time, our method enables incremental scene understanding through progressive scene construction.
• Figure 3: Local 3D Optimization. When a new keyframe $I_j$ arrives, new 3D Gaussians are initialized from its segmented regions using depth information. To maintain semantic consistency with the existing scene representation, KNN correspondence is established between new Gaussian segments and existing scene Gaussians. Each new Gaussian segment is then evaluated using semantic ambiguity ratio $r(m)$ and spatial-semantic coherence $p(m)$ to determine label assignment. Finally, Gaussian semantic confidence is updated using spatial consistency measure and semantic confidence.
• Figure 4: Designed prompts for progressive graph construction, including caption prompt $\mathcal{P}_\text{caption}$, tag prompt $\mathcal{P}_\text{tag}$, and edge prompt $\mathcal{P}_\text{edge}$.
• Figure 5: Qualitative comparison of semantic segmentation on Replica straub2019replica. Details are highlighted with colorful boxes. Our method achieves more precise semantic representation than baselines.