Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An extensive simulation study evaluating the interaction of resampling techniques across multiple causal discovery contexts

Ritwick Banerjee, Bryan Andrews, Erich Kummerfeld

TL;DR

This work addresses how resampling techniques can validate causal discovery results and guide tuning choices. It introduces theoretical links between effective sample size-based resampling and penalty-discount tuning, and validates them through large-scale simulations using BOSS and fGES across ER and scale-free graphs with 20 or 100 variables. The results show no single resampling method dominates, but certain combinations (e.g., ESS with penalty discounts, 90% subsampling with higher penalties) consistently improve precision and calibration and protect against erroneous edges, especially at smaller sample sizes. The findings offer practical guidelines for selecting resampling strategies and tuning parameters in high-dimensional causal discovery, with implications for robustness and reliability in non-experimental settings.

Abstract

Despite the accelerating presence of exploratory causal analysis in modern science and medicine, the available non-experimental methods for validating causal models are not well characterized. One of the most popular methods is to evaluate the stability of model features after resampling the data, similar to resampling methods for estimating confidence intervals in statistics. Many aspects of this approach have received little to no attention, however, such as whether the choice of resampling method should depend on the sample size, algorithms being used, or algorithm tuning parameters. We present theoretical results proving that certain resampling methods closely emulate the assignment of specific values to algorithm tuning parameters. We also report the results of extensive simulation experiments, which verify the theoretical result and provide substantial data to aid researchers in further characterizing resampling in the context of causal discovery analysis. Together, the theoretical work and simulation results provide specific guidance on how resampling methods and tuning parameters should be selected in practice.

An extensive simulation study evaluating the interaction of resampling techniques across multiple causal discovery contexts

TL;DR

This work addresses how resampling techniques can validate causal discovery results and guide tuning choices. It introduces theoretical links between effective sample size-based resampling and penalty-discount tuning, and validates them through large-scale simulations using BOSS and fGES across ER and scale-free graphs with 20 or 100 variables. The results show no single resampling method dominates, but certain combinations (e.g., ESS with penalty discounts, 90% subsampling with higher penalties) consistently improve precision and calibration and protect against erroneous edges, especially at smaller sample sizes. The findings offer practical guidelines for selecting resampling strategies and tuning parameters in high-dimensional causal discovery, with implications for robustness and reliability in non-experimental settings.

Abstract

Despite the accelerating presence of exploratory causal analysis in modern science and medicine, the available non-experimental methods for validating causal models are not well characterized. One of the most popular methods is to evaluate the stability of model features after resampling the data, similar to resampling methods for estimating confidence intervals in statistics. Many aspects of this approach have received little to no attention, however, such as whether the choice of resampling method should depend on the sample size, algorithms being used, or algorithm tuning parameters. We present theoretical results proving that certain resampling methods closely emulate the assignment of specific values to algorithm tuning parameters. We also report the results of extensive simulation experiments, which verify the theoretical result and provide substantial data to aid researchers in further characterizing resampling in the context of causal discovery analysis. Together, the theoretical work and simulation results provide specific guidance on how resampling methods and tuning parameters should be selected in practice.

Paper Structure

This paper contains 23 sections, 18 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Key Contributions:
  4. Outline
  5. Background
  6. Resampling
  7. Directed Acyclic Graphs
  8. Causal Discovery
  9. The Bayesian Information Criterion and Likelihood Ratio Test
  10. Metrics
  11. Brier Scores
  12. Expected Calibration Error
  13. Precision, Recall, and F1-score
  14. Analysis of Resampling Techniques
  15. What, Why and How of using ESS
  16. ...and 8 more sections

Figures (6)

  • Figure 1: Simulation flow diagram
  • Figure 2: Non-uniformity of LRT p-values under the null hypothesis.
  • Figure 3: F1 scores for BOSS and fGES algorithms on ER graphs with 100 variables and average degree 2
  • Figure 4: Precision scores for BOSS algorithm demonstrating the protective effect of resampling against fitting noise
  • Figure 5: Brier scores for BOSS and fGES algorithms demonstrating calibration quality
  • ...and 1 more figures