Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Event-Grounding Graph: Unified Spatio-Temporal Scene Graph from Robotic Observations

Phuoc Nguyen, Francesco Verdoja, Ville Kyrki

TL;DR

Egg introduces a unified spatio-temporal scene graph that grounds observed dynamic events to spatial scene elements, enabling robust natural language querying for robotic observations. It combines a hierarchical spatial graph (3dsg-like) with an event graph, linking events to involved objects via time-stamped edges, and it enables task-driven pruning to form a relevant subgraph for reasoning. Experiments on real-world robotic data show that including both spatial and event components, together with pruning, improves downstream semantic scores and reduces token usage, while video captioning quality remains a bottleneck. The work provides an open-source implementation and a dataset to advance spatio-temporal scene understanding for autonomous robotics, with potential impact on memory-augmented perception, semantic search, and human-robot collaboration.

Abstract

A fundamental aspect for building intelligent autonomous robots that can assist humans in their daily lives is the construction of rich environmental representations. While advances in semantic scene representations have enriched robotic scene understanding, current approaches lack a connection between spatial features and dynamic events; e.g., connecting the blue mug to the event washing a mug. In this work, we introduce the event-grounding graph (EGG), a framework grounding event interactions to spatial features of a scene. This representation allows robots to perceive, reason, and respond to complex spatio-temporal queries. Experiments using real robotic data demonstrate EGG's capability to retrieve relevant information and respond accurately to human inquiries concerning the environment and events within. Furthermore, the EGG framework's source code and evaluation dataset are released as open-source at: https://github.com/aalto-intelligent-robotics/EGG.

Event-Grounding Graph: Unified Spatio-Temporal Scene Graph from Robotic Observations

TL;DR

Egg introduces a unified spatio-temporal scene graph that grounds observed dynamic events to spatial scene elements, enabling robust natural language querying for robotic observations. It combines a hierarchical spatial graph (3dsg-like) with an event graph, linking events to involved objects via time-stamped edges, and it enables task-driven pruning to form a relevant subgraph for reasoning. Experiments on real-world robotic data show that including both spatial and event components, together with pruning, improves downstream semantic scores and reduces token usage, while video captioning quality remains a bottleneck. The work provides an open-source implementation and a dataset to advance spatio-temporal scene understanding for autonomous robotics, with potential impact on memory-augmented perception, semantic search, and human-robot collaboration.

Abstract

A fundamental aspect for building intelligent autonomous robots that can assist humans in their daily lives is the construction of rich environmental representations. While advances in semantic scene representations have enriched robotic scene understanding, current approaches lack a connection between spatial features and dynamic events; e.g., connecting the blue mug to the event washing a mug. In this work, we introduce the event-grounding graph (EGG), a framework grounding event interactions to spatial features of a scene. This representation allows robots to perceive, reason, and respond to complex spatio-temporal queries. Experiments using real robotic data demonstrate EGG's capability to retrieve relevant information and respond accurately to human inquiries concerning the environment and events within. Furthermore, the EGG framework's source code and evaluation dataset are released as open-source at: https://github.com/aalto-intelligent-robotics/EGG.
Paper Structure (19 sections, 4 equations, 4 figures, 1 table)

This paper contains 19 sections, 4 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related works
  3. Problem Formulation
  4. Methodology
  5. egg generation
  6. egg pruning
  7. Implementation
  8. Building from robotic observations
  9. Task-based graph pruning
  10. Evaluation
  11. Experimental setup
  12. Experimental results
  13. The role of each component of
  14. Graph pruning efficiency
  15. Video captioning capability
  16. ...and 4 more sections

Figures (4)

  • Figure 1: We propose egg, a scene representation framework that grounds observed events with involved spatial elements (e.g., objects) to allow robots to respond to complex queries.
  • Figure 2: egg architecture
  • Figure 3: Task-based graph manipulation
  • Figure 4: Summary of failure modes of egg with pruning and ground truth captioning.