Unbiased Video Scene Graph Generation via Visual and Semantic Dual Debiasing

TL;DR

This paper tackles biases in video scene graph generation by explicitly separating and mitigating visual and semantic biases. It introduces VISA, a dual debiasing framework that combines memory-guided temporal integration (MGSM) for visual debiasing with an Iterative Relation Generator (IRG) and Hierarchical Semantics Extractor (HSE) for semantic debiasing, enabling robust extraction of dynamic relationships across frames. The method delivers state-of-the-art unbiased VidSGG results on Action Genome, including substantial gains in SGCLS under Semi Constraint, and demonstrates the importance of integrating visual and semantic cues through memory and context. The work has practical impact for reliable scene understanding in videos and lays groundwork for extending visual–semantic debiasing to broader multimodal perception tasks.

Abstract

Video Scene Graph Generation (VidSGG) aims to capture dynamic relationships among entities by sequentially analyzing video frames and integrating visual and semantic information. However, VidSGG is challenged by significant biases that skew predictions. To mitigate these biases, we propose a VIsual and Semantic Awareness (VISA) framework for unbiased VidSGG. VISA addresses visual bias through memory-enhanced temporal integration that enhances object representations and concurrently reduces semantic bias by iteratively integrating object features with comprehensive semantic information derived from triplet relationships. This visual-semantics dual debiasing approach results in more unbiased representations of complex scene dynamics. Extensive experiments demonstrate the effectiveness of our method, where VISA outperforms existing unbiased VidSGG approaches by a substantial margin (e.g., +13.1% improvement in mR@20 and mR@50 for the SGCLS task under Semi Constraint).

Figures (9)

• Figure 2: Framework of VISA. Given an input video, VISA extracts object features using an off-the-shelf detector. Subsequently, the Memory Guided Sequence Modeling leverages these features to develop robust object representations. Following this, the Iterative Relation Generator employs the object features to predict relationships between subject-object pairs. It is important to note that the Hierarchical Semantic Extractor is not engaged during the initial phase of scene graph generation.
• Figure 3: Structure of Memory Guided Sequence Modeling.
• Figure 4: Structure of Hierarchical Semantics Extractor.
• Figure 5: Qualitative results of our proposed method. Comparison with the open-sourced state-of-the-art model TEMPURA nag2023unbiased under Semi Constraint, closely matching ground truth cong2021spatial. Visualizations show attention intensity and video scene graphs from Frame $t$ to $t+1$. False positive relationships and object classifications are highlighted in purple blocks.
• Figure 6: Comparison of TEMPURA and VISA visual feature representation variance. Visualization of visual feature representation variance across training epochs. The shaded regions represent the standard deviation.