TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Models

TL;DR

TimeCausality introduces a dedicated benchmark to probe temporal-causal reasoning in Vision-Language Models by focusing on irreversible, real-world state changes. It establishes a five-category causal taxonomy and a three-aspect evaluation (awareness, reasoning, inferring) built on a high-quality, human-verified image-pair dataset generated via an LLM-guided inpainting pipeline. Across open-source and closed-source models, the study reveals a pronounced gap: GPT-4o leads on ordering but open-source VLMs struggle with temporal judgments, underscoring the need to incorporate temporal causality into both evaluation and training. The work provides a robust data-generation framework and evaluation prompts, offering a clear path for advancing real-world commonsense and temporal understanding in multimodal models.

Abstract

Reasoning about temporal causality, particularly irreversible transformations of objects governed by real-world knowledge (e.g., fruit decay and human aging), is a fundamental aspect of human visual understanding. Unlike temporal perception based on simple event sequences, this form of reasoning requires a deeper comprehension of how object states change over time. Although the current powerful Vision-Language Models (VLMs) have demonstrated impressive performance on a wide range of downstream tasks, their capacity to reason about temporal causality remains underexplored. To address this gap, we introduce \textbf{TimeCausality}, a novel benchmark specifically designed to evaluate the causal reasoning ability of VLMs in the temporal dimension. Based on our TimeCausality, we find that while the current SOTA open-source VLMs have achieved performance levels comparable to closed-source models like GPT-4o on various standard visual question answering tasks, they fall significantly behind on our benchmark compared with their closed-source competitors. Furthermore, even GPT-4o exhibits a marked drop in performance on TimeCausality compared to its results on other tasks. These findings underscore the critical need to incorporate temporal causality into the evaluation and development of VLMs, and they highlight an important challenge for the open-source VLM community moving forward. Code and Data are available at \href{https://github.com/Zeqing-Wang/TimeCausality }{TimeCausality}.

Figures (8)

• Figure 1: Illustration of our motivation. The content of an image will change over time. For example, a person will age over time, and this change is a time-dependent and irreversible process. Our goal is to evaluate VLM's ability to understand such a causal relationship.
• Figure 2: Causal types in our TimeCausality benchmark. We identify five dimensions of temporal causality: Artificial Processing (AP), Chemical Change (CC), Natural Phenomenon (NP), Environmental Modification (EM), and Physical Change (PC). The image depicting the causal result is generated by modifying the original using an inpainting model. The object marked in RED denotes the primary target of causality.
• Figure 3: Illustration of TimeCausality generation pipeline. First, we use a grounding model with an auto-labeled model to detect and label key objects in the original image. Based on the detected objects, an LLM generates possible irreversible transformations that the object could undergo over time and generates corresponding modification instructions along with reasoning explanations (e.g., turning a pizza into a moldy state). Next, we apply the powerful GPT-4o as the inpainting model to modify the original image according to the generated editing instruction, creating a temporally evolved version of the object within the image. Finally, we pair the original and modified images and incorporate the reasoning and inferring explanation from the instruction step as the ground truth to construct our TimeCausality. All data are manually checked following the instructions provided in Appendix \ref{['sec:human_verification']}.
• Figure 4: Data demo and case study in our TimeCausality.
• Figure 5: Evaluation aspects of our TimeCausality.