Estimating Commonsense Scene Composition on Belief Scene Graphs

TL;DR

This work defines commonsense scene composition as the joint spatial distribution of unseen objects within Belief Scene Graphs, enabling scalable, uncertainty-aware reasoning for indoor robotics. It introduces the CECI framework, with a baseline GCN variant and a neuro-symbolic extension using a language-enabled spatial ontology, to estimate $\hat{L}_\mathcal{N}(C)$ from partially observed scenes. A full data pipeline from semantically annotated Matterport3D scenes to ground-truth heatmaps and augmented Belief Scene Graphs is described, along with a robust evaluation comparing distributional distances and a real-world field test with a legged robot. The results show that incorporating ontology improves distributional alignment and practical capability to infer unseen objects across room types, highlighting practical impact for object localization and task planning under uncertainty in robotics.

Abstract

This work establishes the concept of commonsense scene composition, with a focus on extending Belief Scene Graphs by estimating the spatial distribution of unseen objects. Specifically, the commonsense scene composition capability refers to the understanding of the spatial relationships among related objects in the scene, which in this article is modeled as a joint probability distribution for all possible locations of the semantic object class. The proposed framework includes two variants of a Correlation Information (CECI) model for learning probability distributions: (i) a baseline approach based on a Graph Convolutional Network, and (ii) a neuro-symbolic extension that integrates a spatial ontology based on Large Language Models (LLMs). Furthermore, this article provides a detailed description of the dataset generation process for such tasks. Finally, the framework has been validated through multiple runs on simulated data, as well as in a real-world indoor environment, demonstrating its ability to spatially interpret scenes across different room types.

Figures (5)

• Figure 1: Depiction of the proposed method for commonsense scene composition estimation based on Belief Scene Graphs, where the BSG is an abstract representation of the scene which encompasses building, rooms, objects and blind nodes. The graph information is then encoded as the set of location heatmaps $L^{\prime\prime}_\mathcal{N}(C) \times O_\mathcal{N}$ that are input to the proposed CECI model to output the predicted set of location heatmaps $\hat{L}_\mathcal{N}(C)$.
• Figure 2: Language-enabled spatial ontology.
• Figure 3: Depiction of the process used to construct the layout of a room based on the estimated location heatmaps for each object class. The input and ground truth of each step is also included for comparison.
• Figure 4: Depiction of the final result, where the proposed method is used to estimate the commonsense scene composition of the environment allowing to complete the Belief Scene Graph with the estimated locations.
• Figure 5: The generated 3D scene graph using the spot legged robot in an indoor environment, where the cylinder represents the building, the spheres represent rooms, and cubes represent objects. Alongside, the BSG, the scene pointcloud and the voxel representations of the estimated location heatmaps are shown. On the right, the ground truth location of the Most Relevant Semantic (MRS) for selected object classes are visualized in SoI heatmaps .