Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Belief Scene Graphs: Expanding Partial Scenes with Objects through Computation of Expectation

Mario A. V. Saucedo, Akash Patel, Akshit Saradagi, Christoforos Kanellakis, George Nikolakopoulos

TL;DR

The paper addresses planning under perceptual uncertainty by extending partial 3D scene graphs with Belief Scene Graphs (BSG) that incorporate expectation about unseen objects. A graph-based learning approach, Computation of Expectation based on Correlation Information (CECI), uses a Graph Convolutional Network to learn object distributions from histograms and to generate blind nodes that augment the scene graph. A HM3D-based dataset generation pipeline supports training the CECI model in the absence of a published 3D scene-graph repository. The authors validate the approach in simulated object-search tasks and in real-field experiments with a Spot robot, showing improved planning efficiency and plausible unseen-object predictions, with potential impact on exploration and multi-agent mission planning.

Abstract

In this article, we propose the novel concept of Belief Scene Graphs, which are utility-driven extensions of partial 3D scene graphs, that enable efficient high-level task planning with partial information. We propose a graph-based learning methodology for the computation of belief (also referred to as expectation) on any given 3D scene graph, which is then used to strategically add new nodes (referred to as blind nodes) that are relevant to a robotic mission. We propose the method of Computation of Expectation based on Correlation Information (CECI), to reasonably approximate real Belief/Expectation, by learning histograms from available training data. A novel Graph Convolutional Neural Network (GCN) model is developed, to learn CECI from a repository of 3D scene graphs. As no database of 3D scene graphs exists for the training of the novel CECI model, we present a novel methodology for generating a 3D scene graph dataset based on semantically annotated real-life 3D spaces. The generated dataset is then utilized to train the proposed CECI model and for extensive validation of the proposed method. We establish the novel concept of \textit{Belief Scene Graphs} (BSG), as a core component to integrate expectations into abstract representations. This new concept is an evolution of the classical 3D scene graph concept and aims to enable high-level reasoning for task planning and optimization of a variety of robotics missions. The efficacy of the overall framework has been evaluated in an object search scenario, and has also been tested in a real-life experiment to emulate human common sense of unseen-objects. For a video of the article, showcasing the experimental demonstration, please refer to the following link: https://youtu.be/hsGlSCa12iY

Belief Scene Graphs: Expanding Partial Scenes with Objects through Computation of Expectation

TL;DR

The paper addresses planning under perceptual uncertainty by extending partial 3D scene graphs with Belief Scene Graphs (BSG) that incorporate expectation about unseen objects. A graph-based learning approach, Computation of Expectation based on Correlation Information (CECI), uses a Graph Convolutional Network to learn object distributions from histograms and to generate blind nodes that augment the scene graph. A HM3D-based dataset generation pipeline supports training the CECI model in the absence of a published 3D scene-graph repository. The authors validate the approach in simulated object-search tasks and in real-field experiments with a Spot robot, showing improved planning efficiency and plausible unseen-object predictions, with potential impact on exploration and multi-agent mission planning.

Abstract

In this article, we propose the novel concept of Belief Scene Graphs, which are utility-driven extensions of partial 3D scene graphs, that enable efficient high-level task planning with partial information. We propose a graph-based learning methodology for the computation of belief (also referred to as expectation) on any given 3D scene graph, which is then used to strategically add new nodes (referred to as blind nodes) that are relevant to a robotic mission. We propose the method of Computation of Expectation based on Correlation Information (CECI), to reasonably approximate real Belief/Expectation, by learning histograms from available training data. A novel Graph Convolutional Neural Network (GCN) model is developed, to learn CECI from a repository of 3D scene graphs. As no database of 3D scene graphs exists for the training of the novel CECI model, we present a novel methodology for generating a 3D scene graph dataset based on semantically annotated real-life 3D spaces. The generated dataset is then utilized to train the proposed CECI model and for extensive validation of the proposed method. We establish the novel concept of \textit{Belief Scene Graphs} (BSG), as a core component to integrate expectations into abstract representations. This new concept is an evolution of the classical 3D scene graph concept and aims to enable high-level reasoning for task planning and optimization of a variety of robotics missions. The efficacy of the overall framework has been evaluated in an object search scenario, and has also been tested in a real-life experiment to emulate human common sense of unseen-objects. For a video of the article, showcasing the experimental demonstration, please refer to the following link: https://youtu.be/hsGlSCa12iY
Paper Structure (13 sections, 3 equations, 5 figures, 2 tables)

This paper contains 13 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. PROBLEM FORMULATION
  3. BELIEF SCENE GRAPHS
  4. CECI Model Architecture Design and Optimization
  5. Dataset Generation for CECI Model Training
  6. Incorporating Expectation on Belief Scene Graphs
  7. EXPERIMENTAL EVALUATION
  8. CECI Model Training
  9. Validation Metrics
  10. Qualitative Analysis
  11. Belief Scene Graphs for Object Search Task
  12. Field Test Results for CECI
  13. CONCLUSIONS

Figures (5)

  • Figure 1: Depiction of a high-level task planning process based on Belief Scene Graphs, where the environment is first abstracted into data through perception, in order to build a 3D scene graph representing building, rooms and objects, and then transform into the set of histograms $E^{\prime}_\mathcal{N}(C) \times O^{\prime}_\mathcal{N}$ that are input to the proposed CECI model to output the predicted set of histograms $E^{\prime\prime}_\mathcal{N}(C) \times O^{\prime\prime}_\mathcal{N}$ used to build a Belief Scene Graph with blind nodes and to draw expectation about the location of an artifact, which enables a sequential high-level task planning .
  • Figure 2: Comparison of the error between the predicted number of objects $O^{\prime\prime}_\mathcal{N}$ and the ground truth $O_\mathcal{N}$ per class label on the validation split.
  • Figure 3: Comparison of the correlation matrices for the 45 labeled object classes of (a) the predicted graphs $\mathcal{G}^{\prime\prime}$ and (b) the ground truth $\mathcal{G}$ over the validation split.
  • Figure 4: Comparison of the input, predicted and ground truth (a) histograms and (b) the corresponding probability distributions over the building node of a single environment of the validation split.
  • Figure 5: The Belief Scene Graph generated using the spot legged robot in an indoor environment, where cylinders represent buildings, squares represent rooms, and spheres represent objects and blind nodes.