An Empirical Examination of Balancing Strategy for Counterfactual Estimation on Time Series

TL;DR

This work empirically interrogates the effectiveness of balancing strategies for counterfactual estimation in time series. Across synthetic and semi-synthetic datasets, it shows that representation-based balancing methods often fail to improve, and can even destabilize, multi-step counterfactual predictions under substantial treatment bias, with ERM variants frequently outperforming BRM-based models. The study additionally reveals that temporal dependencies yield value mainly under low bias, while high bias erodes their usefulness, suggesting a need to rethink balancing approaches in temporal causal inference. The findings motivate future work on balancing-prediction trade-offs, more stable balancing mechanisms, and pre-assessment of treatment bias to determine when balancing should be applied.

Abstract

Counterfactual estimation from observations represents a critical endeavor in numerous application fields, such as healthcare and finance, with the primary challenge being the mitigation of treatment bias. The balancing strategy aimed at reducing covariate disparities between different treatment groups serves as a universal solution. However, when it comes to the time series data, the effectiveness of balancing strategies remains an open question, with a thorough analysis of the robustness and applicability of balancing strategies still lacking. This paper revisits counterfactual estimation in the temporal setting and provides a brief overview of recent advancements in balancing strategies. More importantly, we conduct a critical empirical examination for the effectiveness of the balancing strategies within the realm of temporal counterfactual estimation in various settings on multiple datasets. Our findings could be of significant interest to researchers and practitioners and call for a reexamination of the balancing strategy in time series settings.

Figures (12)

• Figure 1: Performance comparison between balanced models (Causal Transformer and CRN) and their ERM variants in multi-step counterfactual estimation on Tumor Growth dataset, the X-axis represents Horizon, Y-axis represents RMSE, lower is better.
• Figure 2: Causal structure in temporal setting, take time step $T-1$ to $T+1$ as an example, where green and red arrows denote the current and time-varying treatment bias, respectively.
• Figure 3: Road map of the experimental evaluation in this study.
• Figure 4: Distribution shape for the two covariates under different strengths of treatment bias on Tumor. (a)-(f) for the first covariate, (g)-(l) for the second covariate.
• Figure 5: Distribution shape for the covariate under different strengths of treatment bias and time-steps on pure synthetic dataset, (a)-(d) for $\gamma=0.2$, (e)-(h) for $\gamma=0.4$.