A Scene Graph Backed Approach to Open Set Semantic Mapping

TL;DR

Open-set semantic mapping is advanced by making a 3D Semantic Scene Graph ($3DSSG$) the backbone of the mapping process, thereby unifying perception and symbolic reasoning in large-scale environments. The approach delivers a GPU-accelerated, incremental pipeline with a three-layer scene graph backend, two-stage data association, and integrated vision-language features to support open-vocabulary queries. It further introduces Incremental Predicate Prediction (IPP) via a heterogeneous Graph Neural Network to infer and propagate semantic relations between open-set objects, enabling richer reasoning. Real-world deployment on the TIAGo platform and evaluation on datasets such as 3RScan demonstrates practical viability, while recognizing current reliance on supervised relational labels and a plan to pursue unsupervised geometric relation extraction for open-world generalization.

Abstract

While Open Set Semantic Mapping and 3D Semantic Scene Graphs (3DSSGs) are established paradigms in robotic perception, deploying them effectively to support high-level reasoning in large-scale, real-world environments remains a significant challenge. Most existing approaches decouple perception from representation, treating the scene graph as a derivative layer generated post hoc. This limits both consistency and scalability. In contrast, we propose a mapping architecture where the 3DSSG serves as the foundational backend, acting as the primary knowledge representation for the entire mapping process. Our approach leverages prior work on incremental scene graph prediction to infer and update the graph structure in real-time as the environment is explored. This ensures that the map remains topologically consistent and computationally efficient, even during extended operations in large-scale settings. By maintaining an explicit, spatially grounded representation that supports both flat and hierarchical topologies, we bridge the gap between sub-symbolic raw sensor data and high-level symbolic reasoning. Consequently, this provides a stable, verifiable structure that knowledge-driven frameworks, ranging from knowledge graphs and ontologies to Large Language Models (LLMs), can directly exploit, enabling agents to operate with enhanced interpretability, trustworthiness, and alignment to human concepts.

Figures (6)

• Figure 1: The heterogeneous graph model for Incremental Predicate Prediction (IPP) from Renz2025-uk. The local graph corresponds to the segments layer or the frame-only local scene graph, the global graph corresponds to the objects layer or the fully integrated and refined graph from the mapping pipeline described in Section \ref{['sec:mapping-approach']}. Dashed edges are between matched objects between the local frame and global map to provide information flow during message passing. While object classes are displayed here for clarity, the actual global scene graph only contains open-set features.
• Figure 2: Graph neural network architecture for Incremental Predicate Prediction (IPP).
• Figure 3: Preliminary results of the mapping pipeline on the small-scale ICL dataset. The segmentation was obtained by comparing each object's CLIP feature with the embeddings of the NYU-40 label set.
• Figure 4: Visualization of local scene graph (yellow) and global scene graph (blue) with inter- and intra-layer edges with a mapped scene from the 3RScan dataset. Predicted edges are integrated into the global graph using ground truth annotations from Wald2020-yj.
• Figure 5: Mapping results and sample queries on real-world data. (a) Successful map integration given the estimated poses. (b) & (c) Visualization of cosine similarity between instances and the respective text queries.