Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Generative Methods for Causal Evaluation via Simulation-Based Inference

Pracheta Amaranath, Vinitra Muralikrishnan, Amit Sharma, David Jensen

Abstract

Generating synthetic datasets that accurately reflect real-world observational data is critical for evaluating causal estimators, but it remains a challenging task. Existing generative methods offer a solution by producing synthetic datasets anchored in the observed data (source data) while allowing variation in key parameters such as the treatment effect and amount of confounding bias. However, it is often unclear which generative methods to use and which values of parameters to choose when generating synthetic datasets. Moreover, existing methods typically require users to provide fixed point estimates of such parameters. This denies users the ability to express uncertainty over both generative methods and parameter values and removes the potential for posterior inference, potentially leading to unreliable estimator comparisons. We introduce simulation-based inference for causal evaluation (SBICE), a framework that treats the generative method and its corresponding generative parameters as uncertain and infers their posterior distribution given a source dataset. Leveraging techniques in simulation-based inference, SBICE identifies suitable generative methods and infers distributions over its parameter configurations to produce synthetic datasets closely aligned with the source data distribution. Empirical results demonstrate that SBICE improves the reliability of estimator evaluations by generating realistic datasets whose causal estimates closely match the estimates of the source data, making it a robust and uncertainty-aware approach to selecting causal estimators.

Improving Generative Methods for Causal Evaluation via Simulation-Based Inference

Abstract

Generating synthetic datasets that accurately reflect real-world observational data is critical for evaluating causal estimators, but it remains a challenging task. Existing generative methods offer a solution by producing synthetic datasets anchored in the observed data (source data) while allowing variation in key parameters such as the treatment effect and amount of confounding bias. However, it is often unclear which generative methods to use and which values of parameters to choose when generating synthetic datasets. Moreover, existing methods typically require users to provide fixed point estimates of such parameters. This denies users the ability to express uncertainty over both generative methods and parameter values and removes the potential for posterior inference, potentially leading to unreliable estimator comparisons. We introduce simulation-based inference for causal evaluation (SBICE), a framework that treats the generative method and its corresponding generative parameters as uncertain and infers their posterior distribution given a source dataset. Leveraging techniques in simulation-based inference, SBICE identifies suitable generative methods and infers distributions over its parameter configurations to produce synthetic datasets closely aligned with the source data distribution. Empirical results demonstrate that SBICE improves the reliability of estimator evaluations by generating realistic datasets whose causal estimates closely match the estimates of the source data, making it a robust and uncertainty-aware approach to selecting causal estimators.

Paper Structure

This paper contains 157 sections, 1 theorem, 49 equations, 52 figures, 22 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Prior Work
  3. Inconsistencies in Generative Methods for Causal Evaluation
  4. Simulation-Based Inference for Causal Evaluation (SBICE)
  5. Simulators.
  6. Priors.
  7. Distance metric.
  8. Posterior inference via SMC-ABC.
  9. Phase 1: Train simulators.
  10. Phase 2: SMC-ABC Posterior Inference.
  11. Phase 3: Generate posterior datasets and compute causal estimates.
  12. Experiments
  13. Bias squared error (BSE).
  14. Aggregating BSE across datasets.
  15. Datasets
  16. ...and 142 more sections

Key Result

proposition 1

Incompatibility of fixed DGP parameters under generative modeling: Let $P(D;\tau^*)$ denote the true distribution of the source data $D = \{X, T, Y\}$, for a binary treatment $T$, consistent with the DGP parameter: $\tau^*$. Let $\hat{P}(D; \tau)$ represent the distribution of datasets generated und

Figures (52)

  • Figure 1: Workflow for simulation-based inference for causal evaluation (SBICE). Simulators corresponding to parametric/non-parametric generative models that generate data similar to the source dataset. Then, using SMC-ABC, we generate posterior datasets that are closer in distance to the source data. These datasets can then be used for reliable and robust benchmarking of causal estimators.
  • Figure 2: Boxplots of estimator bias (estimated ATE $-$ true ATE) across four generative methods for the Lalonde dataset under three DGP parameter setting: (a) Learned ATE, (b) True ATE, and (c) Incorrect ATE. Estimator performance varies substantially across both generative methods and parameter settings, highlighting the sensitivity of causal benchmarking to these choices.
  • Figure 3: BSE for a set of causal estimators for the datasets in Table \ref{['tab:mean-bse-synthetic']}. Lower BSE values for the posterior indicate similarity to the source dataset.
  • Figure 4: Marginal distribution of outcome $Y$ for LinearParam DGP11.
  • Figure 5: ATEs for the generated datasets across generative methods for the three different settings of the constraints.
  • ...and 47 more figures

Theorems & Definitions (2)

  • proposition 1
  • proof