Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IncomeSCM: From tabular data set to time-series simulator and causal estimation benchmark

Fredrik D. Johansson

Abstract

Evaluating observational estimators of causal effects demands information that is rarely available: unconfounded interventions and outcomes from the population of interest, created either by randomization or adjustment. As a result, it is customary to fall back on simulators when creating benchmark tasks. Simulators offer great control but are often too simplistic to make challenging tasks, either because they are hand-designed and lack the nuances of real-world data, or because they are fit to observational data without structural constraints. In this work, we propose a general, repeatable strategy for turning observational data into sequential structural causal models and challenging estimation tasks by following two simple principles: 1) fitting real-world data where possible, and 2) creating complexity by composing simple, hand-designed mechanisms. We implement these ideas in a highly configurable software package and apply it to the well-known Adult income data set to construct the IncomeSCM simulator. From this, we devise multiple estimation tasks and sample data sets to compare established estimators of causal effects. The tasks present a suitable challenge, with effect estimates varying greatly in quality between methods, despite similar performance in the modeling of factual outcomes, highlighting the need for dedicated causal estimators and model selection criteria.

IncomeSCM: From tabular data set to time-series simulator and causal estimation benchmark

Abstract

Evaluating observational estimators of causal effects demands information that is rarely available: unconfounded interventions and outcomes from the population of interest, created either by randomization or adjustment. As a result, it is customary to fall back on simulators when creating benchmark tasks. Simulators offer great control but are often too simplistic to make challenging tasks, either because they are hand-designed and lack the nuances of real-world data, or because they are fit to observational data without structural constraints. In this work, we propose a general, repeatable strategy for turning observational data into sequential structural causal models and challenging estimation tasks by following two simple principles: 1) fitting real-world data where possible, and 2) creating complexity by composing simple, hand-designed mechanisms. We implement these ideas in a highly configurable software package and apply it to the well-known Adult income data set to construct the IncomeSCM simulator. From this, we devise multiple estimation tasks and sample data sets to compare established estimators of causal effects. The tasks present a suitable challenge, with effect estimates varying greatly in quality between methods, despite similar performance in the modeling of factual outcomes, highlighting the need for dedicated causal estimators and model selection criteria.
Paper Structure (24 sections, 12 equations, 3 figures, 7 tables)

This paper contains 24 sections, 12 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related work
  4. Constructing the simulator
  5. On hand-crafted transitions.
  6. The base data set: Adult
  7. Outcome of interest
  8. Intervention of interest
  9. The causal graph
  10. The samplers
  11. Hand-crafted samplers.
  12. Implementation & data set
  13. Benchmark data set
  14. Experiments
  15. Causal effect estimators.
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Overview of the IncomeSCM simulator. All variables (left) are observed at a single time point in the well-known Adult data set, except the Studies variable (node and edges in red). A causal graph is constructed for the initial state (middle-left) and for the transitions, illustrated here by the parents of a single node in the full graph (middle-right). Each variable is associated with an initial sampler and a transition sampler (example to the right) which simulate the trajectory of the variable.
  • Figure 2: Estimated CATE conditioned on education (numeric) by stratifying CATE estimates w.r.t. the full conditioning set of Task 1.
  • Figure 3: Results estimating CATE on IncomeSCM-1.0.CATE, conditioned only on education with the full adjustment set of Task 1. $R^2$ and 95% CI computed from 1000 bootstrap samples uses CATE estimated from counterfactual samples within each education bin as ground truth.