SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation

TL;DR

Dynamic Scene Graph Generation demands understanding evolving spatio-temporal relations in video. SceneLLM converts video to implicit language signals via V2L, encodes spatial structure with SIA, and fuses temporal context with OT, then reasons with a LoRA-finetuned LLM before decoding a dynamic scene graph with a transformer predictor, yielding $G=\{G_\tau\}_{\tau=1}^T$. This approach achieves state-of-the-art results on the Action Genome benchmark, demonstrating that LLMs can function as scene analyzers when provided with language-like representations of visual content. The work integrates VQ-VAE discretization, Chinese-character-inspired spatial encoding, OT-based temporal alignment, and LoRA tuning to bridge vision and language for robust dynamic scene understanding with practical implications for robotics and autonomous systems.

Abstract

Dynamic scenes contain intricate spatio-temporal information, crucial for mobile robots, UAVs, and autonomous driving systems to make informed decisions. Parsing these scenes into semantic triplets <Subject-Predicate-Object> for accurate Scene Graph Generation (SGG) is highly challenging due to the fluctuating spatio-temporal complexity. Inspired by the reasoning capabilities of Large Language Models (LLMs), we propose SceneLLM, a novel framework that leverages LLMs as powerful scene analyzers for dynamic SGG. Our framework introduces a Video-to-Language (V2L) mapping module that transforms video frames into linguistic signals (scene tokens), making the input more comprehensible for LLMs. To better encode spatial information, we devise a Spatial Information Aggregation (SIA) scheme, inspired by the structure of Chinese characters, which encodes spatial data into tokens. Using Optimal Transport (OT), we generate an implicit language signal from the frame-level token sequence that captures the video's spatio-temporal information. To further improve the LLM's ability to process this implicit linguistic input, we apply Low-Rank Adaptation (LoRA) to fine-tune the model. Finally, we use a transformer-based SGG predictor to decode the LLM's reasoning and predict semantic triplets. Our method achieves state-of-the-art results on the Action Genome (AG) benchmark, and extensive experiments show the effectiveness of SceneLLM in understanding and generating accurate dynamic scene graphs.

Figures (7)

• Figure 1: Overview of our proposed SceneLLM framework. In our framework, given an input video signal, we first conduct a Video-to-Language (V2L) Mapping process to obtain the corresponding implicit linguistic signal which is more friendly to LLM. We then perform implicit language reasoning via the fine-tuned LLM with LoRA and employ the SGG predictor to generate the dynamic scene graph.
• Figure 2: Video-to-Language (V2L) Mapping Process: (a) Features of the regions of interest (ROIs) in video frames are extracted via objector detector, and then the learned VQ-VAE discretely quantizes the extracted ROIs features. (b) SIA and OT schemes to embed spatial-temporal information into implicit linguist tokens which are used as inputs of LLM.
• Figure 3: Spatial Information Aggregation for Frame-level Token Generation: (a) Illustration of Spatial Representation of Chinese Characters. (b) Hierarchical Clustering of Discrete Objects' Features for Spatial Correlation. (c) Chinese Character-like Frame-level Token Generation with GCN.
• Figure 4: Update for an optimal codebook $\mathbb{C^+}$ via an optimal transport scheme so that dynamic information can be embedded into implicit linguistic signal.
• Figure 5: Visualization results of our method. All results are given under the SGDET setup. (Zoom in for the best view.)