CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations

TL;DR

CausalVerse introduces a large-scale, high-fidelity CRL benchmark that couples realistic visuals with fully annotated ground-truth causal structures across four domains (static images, dynamic physics, robotics, and traffic). By providing domain–scene–instantiation hierarchies and configurable causal graphs, it enables rigorous identifiability evaluation and stress-testing under unmet assumptions. The paper benchmarks leading CRL methods, revealing gaps between current approaches and ground-truth identifiability, and offers practical guidance for method selection and dataset-driven robustness. This benchmark aims to bridge realism and theoretical guarantees, supporting robust, generalizable causal representations in complex visual environments.

Abstract

Causal Representation Learning (CRL) aims to uncover the data-generating process and identify the underlying causal variables and relations, whose evaluation remains inherently challenging due to the requirement of known ground-truth causal variables and causal structure. Existing evaluations often rely on either simplistic synthetic datasets or downstream performance on real-world tasks, generally suffering a dilemma between realism and evaluative precision. In this paper, we introduce a new benchmark for CRL using high-fidelity simulated visual data that retains both realistic visual complexity and, more importantly, access to ground-truth causal generating processes. The dataset comprises around 200 thousand images and 3 million video frames across 24 sub-scenes in four domains: static image generation, dynamic physical simulations, robotic manipulations, and traffic situation analysis. These scenarios range from static to dynamic settings, simple to complex structures, and single to multi-agent interactions, offering a comprehensive testbed that hopefully bridges the gap between rigorous evaluation and real-world applicability. In addition, we provide flexible access to the underlying causal structures, allowing users to modify or configure them to align with the required assumptions in CRL, such as available domain labels, temporal dependencies, or intervention histories. Leveraging this benchmark, we evaluated representative CRL methods across diverse paradigms and offered empirical insights to assist practitioners and newcomers in choosing or extending appropriate CRL frameworks to properly address specific types of real problems that can benefit from the CRL perspective. Welcome to visit our: Project page:https://causal-verse.github.io/, Dataset:https://huggingface.co/CausalVerse.

Figures (27)

• Figure 1: The CausalVerse dataset. CausalVerse is organized with a hierarchical domain-scene-instance structure. We showcase four diverse domains: static generation, physical simulation, robotic manipulation, and traffic analysis. Within each domain, multiple scenes are designed to reflect distinct causal variables and structures. Sample instances are generated using simulation tools such as Unreal Engine 4, with each instance grounded in a predefined causal graph.
• Figure 2: Data examples of the dataset across four domains. The Static Image Generation domain features the human images in diverse indoor environments. The Dynamic Physical Simulations domain describes different physical processes like spring compression, light refraction, and projectile motion. The Traffic Situation Analysis domain shows the traffic situations under different cities, scenes, and conditions. The Robotic Manipulations domain offers six scenes in which a robot operates in different indoor settings and performs various tasks.
• Figure 3: Latent variable numbers across all scenes in CausalVerse. please zoom in for details, like scene name.
• Figure 4: Sample statistics across all scenes in CausalVerse. The pie chart illustrates the relative data scale for all scenes, where the radius length reflects the number of images/videos (rather than frames), and colors indicate different domains. For example, the scene Cylinder Compressing Spring contains 40k images.
• Figure 5: Data construction pipeline illustrated using the dynamic physical simulation domain. Starting from a task type such as dynamic physical simulation, we first identify candidate scenes and collect relevant causal variables and structures using human domain expertise. Instances are then generated using a rendering engine to ensure alignment with the designed causal structures.