CAUSAL3D: A Comprehensive Benchmark for Causal Learning from Visual Data

TL;DR

CAUSAL3D addresses the lack of benchmarks for inferring latent causality from visual data by pairing structured tabular data with tightly aligned 3D visuals rendered in Blender, enabling controlled causal evaluation. It comprises 19 datasets spanning 2–5 variables across physically consistent and hypothetical scenes and supports three tasks: causal discovery from tabular data, causal representation learning from images, and causal discovery with intervention from few images, including interventions via the do-operator $do(X=x)$. Across traditional causal discovery methods, causal representation learning models, and LLM/VLM-based approaches, performance declines as causal graphs grow more complex and hypothetical knowledge is absent. Causal3D offers a foundation for advancing causal reasoning in CV and supports trustworthy AI in safety-critical domains, with dataset release and evaluation tools to foster future research.

Abstract

True intelligence hinges on the ability to uncover and leverage hidden causal relations. Despite significant progress in AI and computer vision (CV), there remains a lack of benchmarks for assessing models' abilities to infer latent causality from complex visual data. In this paper, we introduce \textsc{\textbf{Causal3D}}, a novel and comprehensive benchmark that integrates structured data (tables) with corresponding visual representations (images) to evaluate causal reasoning. Designed within a systematic framework, Causal3D comprises 19 3D-scene datasets capturing diverse causal relations, views, and backgrounds, enabling evaluations across scenes of varying complexity. We assess multiple state-of-the-art methods, including classical causal discovery, causal representation learning, and large/vision-language models (LLMs/VLMs). Our experiments show that as causal structures grow more complex without prior knowledge, performance declines significantly, highlighting the challenges even advanced methods face in complex causal scenarios. Causal3D serves as a vital resource for advancing causal reasoning in CV and fostering trustworthy AI in critical domains.

Figures (39)

• Figure 1: The proposed Causal3D dataset. We display 8 real-world scenes (11 hypothetical scenes are in the Appendix \ref{['App:Dataset']}). We focus on 3 scenes: springs, parabolas, and water flow. 1) The blue block represents multi-view images of each scene, offering four different views. The first row shows virtual backgrounds, while the second row shows real backgrounds, with the same view in each column; 2) The green block provides textual descriptions; 3) The yellow block represents the causal graphs for each scene, along with the meanings of each variable in the graphs; 4) The pink block shows the structural equations (i.e., functions describing causal relations) for each scene. The bottom row briefly presents an overview of the remaining 5 real-world scenes and their corresponding causal graphs, including reflection, seesaw, convex lens, magnet, and pendulum. Detailed information on these 5 scenes can be found in the Appendix \ref{['App:Dataset']}.
• Figure 2: Illustration of data construction pipeline.
• Figure 3: Results of different causal discovery methods from tabular data on realistic/hypothetical scenes.
• Figure 4: Examples of the 4 causal representation learning model results. In each scene, the 3 columns show: 1) original images, 2) Do(Cau): results after intervening on a "cause" variable, and 3) Do(Res): after intervening on a "result" variable.
• Figure 6: We select two scenes, Spring and Parabola. Using the F1 score as metric, we assess VLM performance in causal discovery. The best and worst views are highlighted to demonstrate the impact of different perspectives. To analyze the effect of multi-view vs. single-view images, we average the performance across 9 individual views in each scene and compare it with the overall multi-view performance. More detailed version is in Appendix \ref{['APP:Experiment']}.