MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoning

TL;DR

This work tackles the gap in video causal reasoning by introducing Multi-Event Causal Discovery (MECD), a task that seeks complete event-level causal graphs across long videos. The proposed Video Granger Causality Model (VGCM) applies an Event Causality Test, leveraging mask-based predictions, front-door and counterfactual causal inference, and context chain reasoning to address causality confounding and illusory causality, while enabling efficient non-regressive complete-graph reasoning. A new MECD dataset, MECD+, aggregates long-form videos from ActivityNet Captions, EgoSchema, and NExTVideo with ground-truth causality and complete-causality annotations validated via GPT-4 and human refinement. Empirically, VGCM achieves state-of-the-art performance on causal chain and complete causal-graph discovery, outperforms leading LLMs and VideoLLMs, and improves downstream tasks such as VQA and Event Prediction when guided by the inferred causal graphs. The work demonstrates practical impact for robust video understanding and provides a scalable framework for event-level causal discovery in complex visual narratives.

Abstract

Video causal reasoning aims to achieve a high-level understanding of videos from a causal perspective. However, it exhibits limitations in its scope, primarily executed in a question-answering paradigm and focusing on brief video segments containing isolated events and basic causal relations, lacking comprehensive and structured causality analysis for videos with multiple interconnected events. To fill this gap, we introduce a new task and dataset, Multi-Event Causal Discovery (MECD). It aims to uncover the causal relations between events distributed chronologically across long videos. Given visual segments and textual descriptions of events, MECD identifies the causal associations between these events to derive a comprehensive and structured event-level video causal graph explaining why and how the result event occurred. To address the challenges of MECD, we devise a novel framework inspired by the Granger Causality method, incorporating an efficient mask-based event prediction model to perform an Event Granger Test. It estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to mitigate challenges in MECD like causality confounding and illusory causality. Additionally, context chain reasoning is introduced to conduct more robust and generalized reasoning. Experiments validate the effectiveness of our framework in reasoning complete causal relations, outperforming GPT-4o and VideoChat2 by 5.77% and 2.70%, respectively. Further experiments demonstrate that causal relation graphs can also contribute to downstream video understanding tasks such as video question answering and video event prediction.

Figures (19)

• Figure 1: (a): Illustration of Multi-Event Causal Discovery Task, where a complete causal graph of traffic surveillance videos is shown. Our task aims to determine whether a causal relation exists between events and outputs a structured causal graph in (b). (c): Example of causality confounding. (d)&(e): Illustration of illusory causality.
• Figure 2: Constitute of the MECD Dataset. We present 5 main video categories and the verb. word cloud of the dataset.
• Figure 3: Statistics of the MECD Dataset. As shown in Figures (a1) and (a2), our dataset mainly analyzes videos that are longer than 120 seconds and contain five or more events. As shown in Figures (b1) and (b2), the proportion of causal and non-causal relations between events in the video of MECD is relatively balanced, moreover emphasizing the existence of causal relations between adjacent events.
• Figure 4: Annotation Examples of the MECD Dataset. Newly annotated attributes "causality", "complete causality" and the formerly existing attributes "sentences", "duration", and "timestamps" are shown along with the video frames.
• Figure 5: Video Granger Causality Model. Two data streams {${\boldsymbol V}^p, {\boldsymbol C}^{p}$} (with certain $e_k$) and {${\boldsymbol V}^m_k, {\boldsymbol C}^m_k$} (without $e_k$) serve as input, video and text embeddings are concatenated after being separately embedded. During dual-branch processing, these two streams are encoded, and further multi-modal information interactions are conducted. The final mask-based prediction module conducts reasoning with two decoded prediction features and ground truth features embedded by {${\boldsymbol v}_N, {\boldsymbol c}_N$}. Further causal inference is conducted for mitigating causality confounding and illusory as in Sec. \ref{['causal']}, and random context masking and non-regressive complete graph reasoning are introduced for robust and efficient reasoning as in Sec. \ref{['sec: context']} and Sec. \ref{['sec: complete']}