MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning

TL;DR

This work introduces Multi-Event Causal Discovery (MECD), a task and dataset for uncovering causal relations among chronologically distributed events in long videos, producing a structured event-level causal diagram. The authors propose VGCM, a Video Granger Causality Model that uses a Granger-inspired Event Causality Test with a mask-based event-prediction framework and a multi-modal encoder-decoder backbone, augmented by front-door adjustment and counterfactual inference to mitigate confounding and illusory causality. The approach achieves state-of-the-art results on MECD, outperforming strong LLM-based baselines (e.g., GPT-4o, VideoLLaVA) by substantial margins and demonstrating robustness, efficiency, and generalization across related tasks. This work advances video understanding by enabling explicit, testable causal diagrams over multiple events, with potential applications in long-form video analysis and reasoning tasks.

Abstract

Video causal reasoning aims to achieve a high-level understanding of video content from a causal perspective. However, current video reasoning tasks are limited in scope, primarily executed in a question-answering paradigm and focusing on short videos containing only a single event and simple causal relationships, lacking comprehensive and structured causality analysis for videos with multiple events. To fill this gap, we introduce a new task and dataset, Multi-Event Causal Discovery (MECD). It aims to uncover the causal relationships between events distributed chronologically across long videos. Given visual segments and textual descriptions of events, MECD requires identifying the causal associations between these events to derive a comprehensive, structured event-level video causal diagram explaining why and how the final result event occurred. To address MECD, we devise a novel framework inspired by the Granger Causality method, using an efficient mask-based event prediction model to perform an Event Granger Test, which estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to address challenges in MECD like causality confounding and illusory causality. Experiments validate the effectiveness of our framework in providing causal relationships in multi-event videos, outperforming GPT-4o and VideoLLaVA by 5.7% and 4.1%, respectively.

Figures (13)

• Figure 1: (a): Illustration of Multi-Event Causal Discovery Task, where the 3rd and 5th premise events account for the occurrence of the final event. The objective of our task is to determine whether a causal relation exists between events and outputs a structured causal diagram. (c): Example of causality confounding. (d)&(e): Illustration of illusory causality.
• Figure 2: Constitute of MECD dataset. In (a1), we present 5 main video categories of the dataset. The word cloud is also summarized for video types. In (a2), the left chart indicates the impact of positions of events on their causality where we find the second last event tends to be more significant; while the right chart plots the number of events in videos.
• Figure 3: Video Granger Causality Model. Two data streams ${\boldsymbol V}^p$ and ${\boldsymbol V}^m_k$ serve as input, video and text embeddings are concatenated after being separately embedded. The VGCM incorporates two classifiers, the caption head takes the unmasked stream to accomplish the event-predicting task, while the relation head discovers the causal relations with two embedding streams.
• Figure 4: Causal Effect of the Adjacent Events and Causality Diagram. (a1) shows the causality of the third event analyzed, the red causal effect needs to be compensated while the green needs to be mitigated. (a2) shows the causal inference methods corresponding to the two causal effects.
• Figure 5: Dataset robustness. Accuracy decreases slightly when increasing noise, and increases slowly when increasing the training data.