Causality Model for Semantic Understanding on Videos

TL;DR

The work identifies data imbalance and distribution shifts as key bottlenecks in semantic video understanding and introduces a causal-modeling toolkit to address VidVRD and VideoQA. IVRD mitigates long-tail bias in relation detection by learning predicate prototypes and applying do-calculus–style interventions on subject–object pairs, improving tail-predicate prediction and zero-shot generalization. IGV and its extension EIGV frame video-question reasoning as invariant (frame/environment) and equivariant processes, grounding the causal scene to robustify VideoQA while enhancing interpretability through visual explanations. STR and the TranSTR architecture tackle complex, long videos with many objects by adaptively selecting question-critical frames and objects, enabling more reliable reasoning and better answer decoding. Collectively, these models advance robust, interpretable causal understanding in video tasks, with practical implications for real-world video analysis under distribution shifts and environmental perturbations.

Abstract

After a decade of prosperity, the development of video understanding has reached a critical juncture, where the sole reliance on massive data and complex architectures is no longer a one-size-fits-all solution to all situations. The presence of ubiquitous data imbalance hampers DNNs from effectively learning the underlying causal mechanisms, leading to significant performance drops when encountering distribution shifts, such as long-tail imbalances and perturbed imbalances. This realization has prompted researchers to seek alternative methodologies to capture causal patterns in video data. To tackle these challenges and increase the robustness of DNNs, causal modeling emerged as a principle to discover the true causal patterns behind the observed correlations. This thesis focuses on the domain of semantic video understanding and explores the potential of causal modeling to advance two fundamental tasks: Video Relation Detection (VidVRD) and Video Question Answering (VideoQA).

Figures (29)

• Figure 1: Example of how rare cases in semantic video understanding tasks are distracted by (a) object-relation correlation, and (b) environment-answer correlation.
• Figure 2: The research tree for towards causal model for Semantic Understanding on Videos. In VidVRD, we study the long tail issue in relation detection that discovers more rare but informative "tail" relations. In VideoQA, we first address frame-level shallow patterns induced by answer-environment spurious correlation and introduce two model-agnostic learning schemes: invariant grounding and equivariant grounding. Following this line, we extend a more efficient invariant grounding by instantiating a transformer-based design. Finally, we target object-level shallow patterns and manage a performance boost on complex VideoQA.
• Figure 3: An example of SCM.
• Figure 4: (a) The pipeline of the VidVRD task. Relations are separately detected in each short segment and merged afterwards. (b) denotes the long-tailed distributions of relations in ImageNet-VidVRD (b) and Vidor (c). Only 20 predicates are depicted to avoid cluttering.
• Figure 5: A brief overview of the predicate prediction in our IVRD based on an intervention mechanism. $\oplus$ denotes element-wise add operation and "fc" denotes a fully-connected layer. (some operations are omitted for simplicity.)