DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation

TL;DR

DIFFVSGG tackles real-time video scene graph generation by reframing VSGG as an online, diffusion-driven graph denoising task. It unifies object detection, bounding-box regression, and predicate prediction within a single latent embedding and performs frame-by-frame reverse diffusion, guided by past frames and motion cues, to continuously evolve the scene graph. A memory bank stores object trajectories, enabling robust temporal reasoning while maintaining low GPU memory usage. The method achieves state-of-the-art results on Action Genome across PredCLS, SGCLS, and SGDET, and ablations corroborate the benefits of latent diffusion modeling, temporal prompting, and motion-aware denoising. This diffusion-based online approach offers a scalable pathway for real-world video understanding tasks requiring fine-grained relational reasoning over time.

Abstract

Top-leading solutions for Video Scene Graph Generation (VSGG) typically adopt an offline pipeline. Though demonstrating promising performance, they remain unable to handle real-time video streams and consume large GPU memory. Moreover, these approaches fall short in temporal reasoning, merely aggregating frame-level predictions over a temporal context. In response, we introduce DIFFVSGG, an online VSGG solution that frames this task as an iterative scene graph update problem. Drawing inspiration from Latent Diffusion Models (LDMs) which generate images via denoising a latent feature embedding, we unify the decoding of object classification, bounding box regression, and graph generation three tasks using one shared feature embedding. Then, given an embedding containing unified features of object pairs, we conduct a step-wise Denoising on it within LDMs, so as to deliver a clean embedding which clearly indicates the relationships between objects. This embedding then serves as the input to task-specific heads for object classification, scene graph generation, etc. DIFFVSGG further facilitates continuous temporal reasoning, where predictions for subsequent frames leverage results of past frames as the conditional inputs of LDMs, to guide the reverse diffusion process for current frames. Extensive experiments on three setups of Action Genome demonstrate the superiority of DIFFVSGG.

Figures (8)

• Figure 1: (a) Existing VSGG solutions typically adopt an offline training pipeline, dividing the problem into various components: object detection, temporal association, and contextual aggregation. (b) DiffVsgg introduces a new paradigm that performs spatial-temporal reasoning directly as each frame is processed sequentially, enabling progressive, online updates to the scene graph.
• Figure 1: A diagram illustrating step-by-step reasoning process of DiffVsgg.
• Figure 2: Overview of our proposed inter-object relationship learning strategy using latent diffusion models.
• Figure 2: More visual comparison with feng2023exploiting in different time steps.
• Figure 3: Overview of our proposed temporal prompting strategy.