DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes

TL;DR

DyGEnc tackles the challenge of reasoning over dynamic environments by encoding a sequence of textual scene graphs into compact tokens that ground a large language model’s reasoning. The method uses a graph encoder and a sequence encoder (Q-Former) to produce fixed-size latent tokens, which are fine-tuned into an LLM via a parameter-efficient adapter, enabling accurate question answering about histories of human-object interactions. Across STAR and AGQA benchmarks, DyGEnc achieves substantial improvements over vision-based baselines and demonstrates practical viability in a robotic setting, including subgraph retrieval to enable real-time inference. This work provides a scalable, compression-friendly memory mechanism for long-horizon robotic reasoning and offers code to facilitate adoption in real-world systems.

Abstract

The analysis of events in dynamic environments poses a fundamental challenge in the development of intelligent agents and robots capable of interacting with humans. Current approaches predominantly utilize visual models. However, these methods often capture information implicitly from images, lacking interpretable spatial-temporal object representations. To address this issue we introduce DyGEnc - a novel method for Encoding a Dynamic Graph. This method integrates compressed spatial-temporal structural observation representation with the cognitive capabilities of large language models. The purpose of this integration is to enable advanced question answering based on a sequence of textual scene graphs. Extended evaluations on the STAR and AGQA datasets indicate that DyGEnc outperforms existing visual methods by a large margin of 15-25% in addressing queries regarding the history of human-to-object interactions. Furthermore, the proposed method can be seamlessly extended to process raw input images utilizing foundational models for extracting explicit textual scene graphs, as substantiated by the results of a robotic experiment conducted with a wheeled manipulator platform. We hope that these findings will contribute to the implementation of robust and compressed graph-based robotic memory for long-horizon reasoning. Code is available at github.com/linukc/DyGEnc.

Figures (5)

• Figure 1: DyGEnc compactly encodes a dynamic graph (sequence of textual scene graphs) of a changing environment in a few tokens. The resulting representation is then utilized by an aligned large language model for situated logical reasoning and question answering.
• Figure 2: Overview of the DyGEnc pipeline. Given a dynamic scene graph - a sequence of textual scene graphs, where nodes and edges carry attributes encoded by a pre-trained text encoder, we first pass each encoded graph through a graph neural network to generate an aggregated graph token. To preserve temporal information, each graph token is enriched with a positional encoding. Then Q-Former module is applied to capture temporal relations, producing a compact sequence representation in query tokens. Finally, a multilayer perceptron projects these tokens into a large language model’s embedding space, with special tokens marking the start and end of the graph soft prompt. Thus LLM can ground its reasoning based on sensory input.
• Figure 3: Example of cross-attention visualization from Q-Former sequence encoder on STAR benchmark for the text query "Which object did the person sit at after they closed the refrigerator?". We draw cross-attention of 1 Q-Former latent query token to each input graph embedding where 8 blocks represent 2 layer with 4 heads in each. Brighter color represents more model attention.
• Figure 4: Example of cross-attention visualization from Q-Former sequence encoder on STAR benchmark for the text query "Which object did the person throw before they held the dish?". We draw cross-attention of 1 Q-Former latent query token to each input graph embedding where 8 blocks represent 2 layer with 4 heads in each. Brighter color represents more model attention.
• Figure 5: Illustration for a robotic experiment setup - to the left: mobile platform Husky with UR5 manipulator equipped to perform MOVE-AND-PICK task based on DyGEnc output, to the right - general scene overviews from our DRobot benchmark.