Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Robustness of Differentiable Causal Discovery in Misspecified Scenarios

Huiyang Yi, Yanyan He, Duxin Chen, Mingyu Kang, He Wang, Wenwu Yu

TL;DR

The paper tackles the problem of reliably discovering causal graphs when model assumptions are violated in real data. It conducts a large-scale benchmark of $12$ causal discovery methods across $8$ misspecification scenarios, covering linear and nonlinear data, with metrics such as $SHD$ and $SID$ guiding evaluation. The key finding is that differentiable causal discovery methods, especially nonlinear variants like $NOTEARS$-$MLP$ and $DAGMA$, exhibit strong robustness in most misspecifications, though scale variation remains a challenging case; the work also offers theoretical explanations grounded in variance-based identifiability results. Practically, the study provides a comprehensive evaluation standard and motivates further development of scale-invariant loss formulations to enhance applicability in real-world data.

Abstract

Causal discovery aims to learn causal relationships between variables from targeted data, making it a fundamental task in machine learning. However, causal discovery algorithms often rely on unverifiable causal assumptions, which are usually difficult to satisfy in real-world data, thereby limiting the broad application of causal discovery in practical scenarios. Inspired by these considerations, this work extensively benchmarks the empirical performance of various mainstream causal discovery algorithms, which assume i.i.d. data, under eight model assumption violations. Our experimental results show that differentiable causal discovery methods exhibit robustness under the metrics of Structural Hamming Distance and Structural Intervention Distance of the inferred graphs in commonly used challenging scenarios, except for scale variation. We also provide the theoretical explanations for the performance of differentiable causal discovery methods. Finally, our work aims to comprehensively benchmark the performance of recent differentiable causal discovery methods under model assumption violations, and provide the standard for reasonable evaluation of causal discovery, as well as to further promote its application in real-world scenarios.

The Robustness of Differentiable Causal Discovery in Misspecified Scenarios

TL;DR

The paper tackles the problem of reliably discovering causal graphs when model assumptions are violated in real data. It conducts a large-scale benchmark of causal discovery methods across misspecification scenarios, covering linear and nonlinear data, with metrics such as and guiding evaluation. The key finding is that differentiable causal discovery methods, especially nonlinear variants like - and , exhibit strong robustness in most misspecifications, though scale variation remains a challenging case; the work also offers theoretical explanations grounded in variance-based identifiability results. Practically, the study provides a comprehensive evaluation standard and motivates further development of scale-invariant loss formulations to enhance applicability in real-world data.

Abstract

Causal discovery aims to learn causal relationships between variables from targeted data, making it a fundamental task in machine learning. However, causal discovery algorithms often rely on unverifiable causal assumptions, which are usually difficult to satisfy in real-world data, thereby limiting the broad application of causal discovery in practical scenarios. Inspired by these considerations, this work extensively benchmarks the empirical performance of various mainstream causal discovery algorithms, which assume i.i.d. data, under eight model assumption violations. Our experimental results show that differentiable causal discovery methods exhibit robustness under the metrics of Structural Hamming Distance and Structural Intervention Distance of the inferred graphs in commonly used challenging scenarios, except for scale variation. We also provide the theoretical explanations for the performance of differentiable causal discovery methods. Finally, our work aims to comprehensively benchmark the performance of recent differentiable causal discovery methods under model assumption violations, and provide the standard for reasonable evaluation of causal discovery, as well as to further promote its application in real-world scenarios.
Paper Structure (47 sections, 37 equations, 8 figures, 32 tables)

This paper contains 47 sections, 37 equations, 8 figures, 32 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Task formulation
  4. Structure identifiability
  5. Differentiable score-based causal discovery
  6. Experimental design
  7. Synthetic datasets
  8. Model assumption violation scenarios
  9. Data generation
  10. Methods
  11. Hyperparameter settings
  12. Evaluation metrics
  13. Critical experimental results and insights
  14. Current methods' performance in misspecified scenarios
  15. Discussion on CAM performance
  16. ...and 32 more sections

Figures (8)

  • Figure 1: Experimental results under the linear, nonlinear, and mlp settings for both the vanilla scenario and the eight misspecified scenarios. SHD (the lower the better) and SID (the lower the better) are evaluated over 10 trials on the 10 nodes $\mathrm{ER}$-2 graphs. For the differentiable causal discovery method, we present only the optimal results. As the nonlinear settings in Figure \ref{['fig:nonlinear_ER2_10_SHD']} and Figure \ref{['fig:nonlinear_ER2_10_SID']} are more favorable to CAM, we conduct a more reasonable evaluation of CAM and differentiable causal discovery under the MLP setting (Section \ref{['sec:discussion cam']}).
  • Figure 2: Experimental results under the linear and nonlinear $\mathrm{ER}$-2 graphs of 20 nodes. SHD (the lower the better) and SID (the lower the better) are evaluated over 10 trials. For the differentiable causal discovery method, we present only the optimal results. As the nonlinear settings in Figure \ref{['fig:nonlinear_ER2_20_SHD']} and Figure \ref{['fig:nonlinear_ER2_20_SID']} are more favorable to CAM, we conduct a more reasonable evaluation of CAM and differentiable causal discovery under the MLP setting (Section \ref{['sec:discussion cam']}).
  • Figure 3: Experimental results under the linear and nonlinear $\mathrm{ER}$-4 graphs of 10 nodes. SHD (the lower the better) and SID (the lower the better) are evaluated over 10 trials. For the differentiable causal discovery method, we present only the optimal results. As the nonlinear settings in Figure \ref{['fig:nonlinear_ER4_10_SHD']} and Figure \ref{['fig:nonlinear_ER4_10_SID']} are more favorable to CAM, we conduct a more reasonable evaluation of CAM and differentiable causal discovery under the MLP setting (Section \ref{['sec:discussion cam']}).
  • Figure 4: Experimental results under the linear and nonlinear $\mathrm{ER}$-4 graphs of 20 nodes. SHD (the lower the better) and SID (the lower the better) are evaluated over 10 trials. For the differentiable causal discovery method, we present only the optimal results. As the nonlinear settings in Figure \ref{['fig:nonlinear_ER4_20_SHD']} and Figure \ref{['fig:nonlinear_ER4_20_SID']} are more favorable to CAM, we conduct a more reasonable evaluation of CAM and differentiable causal discovery under the MLP setting (Section \ref{['sec:discussion cam']}).
  • Figure 5: Experimental results under the linear and nonlinear $\mathrm{SF}$-2 graphs of 10 nodes. SHD (the lower the better) and SID (the lower the better) are evaluated over 10 trials. For the differentiable causal discovery method, we present only the optimal results. As the nonlinear settings in Figure \ref{['fig:nonlinear_SF2_10_SHD']} and Figure \ref{['fig:nonlinear_SF2_10_SID']} are more favorable to CAM, we conduct a more reasonable evaluation of CAM and differentiable causal discovery under the MLP setting (Section \ref{['sec:discussion cam']}).
  • ...and 3 more figures