VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation

TL;DR

VOST-SGG introduces a VLM-aided, one-stage spatio-temporal scene graph generator that grounds decoder queries with instance-specific semantic priors from vision-language models and augments predicate reasoning with a multimodal feature bank. The dual-source query initialization separates what to attend to from where to attend, enabling semantically informed and spatially stable attention, while the multi-modal bank fuses visual, textual, and spatial cues for robust predicate classification. Through extensive experiments on Action Genome, VOST-SGG achieves state-of-the-art results, particularly boosting performance on rare predicates and improving long-tail robustness, with ablations confirming the benefits of both core innovations. The work demonstrates how integrating VLM priors and multimodal reasoning into DETR-based ST-SGG can substantially enhance interpretable video understanding and downstream reasoning tasks.

Abstract

Spatio-temporal scene graph generation (ST-SGG) aims to model objects and their evolving relationships across video frames, enabling interpretable representations for downstream reasoning tasks such as video captioning and visual question answering. Despite recent advancements in DETR-style single-stage ST-SGG models, they still suffer from several key limitations. First, while these models rely on attention-based learnable queries as a core component, these learnable queries are semantically uninformed and instance-agnostically initialized. Second, these models rely exclusively on unimodal visual features for predicate classification. To address these challenges, we propose VOST-SGG, a VLM-aided one-stage ST-SGG framework that integrates the common sense reasoning capabilities of vision-language models (VLMs) into the ST-SGG pipeline. First, we introduce the dual-source query initialization strategy that disentangles what to attend to from where to attend, enabling semantically grounded what-where reasoning. Furthermore, we propose a multi-modal feature bank that fuses visual, textual, and spatial cues derived from VLMs for improved predicate classification. Extensive experiments on the Action Genome dataset demonstrate that our approach achieves state-of-the-art performance, validating the effectiveness of integrating VLM-aided semantic priors and multi-modal features for ST-SGG. We will release the code at https://github.com/LUNAProject22/VOST.

Figures (13)

• Figure 1: Comparison between (a) Previous work and, (b) Our proposed VOST-SGG framework.
• Figure 2: Overall pipeline of our VOST-SGG framework. Subject-object and predicate queries attend to visual and VLM-aided multi-modal features respectively, followed by prediction heads to generate the final spatio-temporal scene graph. Blue, Red and Green lines depict the textual, visual and spatial embeddings.
• Figure 3: Dual source query initialization process.
• Figure 4: Embeddings derived from multi-modal feature bank, and the predicate decoder.
• Figure 5: Per-predicate mR@10 (PredCLS, with constraint). We compare VOST-SGG to the single-stage OED oed and multi-stage TEMPURA tempura. Predicates are sorted in descending order by frequency from left to right.