Reevaluating Causal Estimation Methods with Data from a Product Release

TL;DR

The study provides an empirical validation of observational causal inference methods using paired experimental and observational data from a large-scale product feature rollout. It demonstrates that with overlap trimming, cross-fitting, nuisance-model ensembling, and a mix of estimators (including $AIPW$ and CATE meta-learners), observational estimates can closely match the experimental $ATE$ for a continuous outcome, but fail to recover the ground truth for a binary outcome due to unobserved confounding. The work highlights the robustness of doubly robust estimators and the critical role of nuisance-function estimation, while also showing the limits of unconfoundedness in practice and the importance of sensitivity analyses. It also extends the validation to the LaLonde benchmark, reinforcing the value of careful design-stage choices and offering practical guidance for credible causal inference in high-dimensional settings.

Abstract

Recent developments in causal machine learning methods have made it easier to estimate flexible relationships between confounders, treatments and outcomes, making unconfoundedness assumptions in causal analysis more palatable. How successful are these approaches in recovering ground truth baselines? In this paper we analyze a new data sample including an experimental rollout of a new feature at a large technology company and a simultaneous sample of users who endogenously opted into the feature. We find that recovering ground truth causal effects is feasible -- but only with careful modeling choices. Our results build on the observational causal literature beginning with LaLonde (1986), offering best practices for more credible treatment effect estimation in modern, high-dimensional datasets.

Figures (19)

• Figure 1: Estimated Propensity Score Distributions Across Samples. Panel (a) shows the distribution of cross-fitted propensity scores estimated on the experimental sample, reflecting the randomized assignment—most scores cluster around the overall treatment rate of 2.64%. Panel (b) shows the distribution of cross-fitted propensity score estimated on the observational sample, where greater separation between treated and untreated groups is evident, consistent with endogenous selection into treatment. Panel (c) plots the observational propensity model $\hat{e}(\cdot)$ evaluated on both the entire observational and entire experimental samples. The complete distributional overlap in Panel (c), coupled with the same range of scores as seen in Panel (b), indicates that the experimental and observational samples cover comparable covariate regions, allowing trimming procedures to be applied consistently across both datasets.
• Figure 2: ATE Estimates for Continuous Outcome (Trimmed Sample): The figure displays ATE estimates and their 95% confidence intervals for five estimators: regression adjustment (Reg), outcome modeling (OM), inverse probability weighting (IPW), propensity score matching (PSM), and doubly robust estimation (DR). After trimming, careful nuisance tuning, and model ensembling, all estimators align closely with the experimental benchmark.
• Figure 3: ATE Estimates for Continuous Outcome (Untrimmed Sample): The figure displays ATE estimates and their 95% confidence intervals for five different estimators: regression adjustment (Reg), outcome modeling (OM), inverse probability weighting (IPW), propensity score matching (PSM), and doubly robust estimation (DR). The black dashed line and associated shaded band represent the naive difference in means and its 95% confidence interval, while the blue dashed line and associated band show the experimental benchmark and its 95% CI. Without trimming, estimates are more dispersed, with only DR encompassing the point estimate of the benchmark.
• Figure 4: DR Estimates Across Different First-Stage Models (Trimmed Sample): This figure compares ATE estimates from doubly robust estimators using a range of flexible and rigid first-stage models for outcome and treatment modeling. Both tuned and sklearn default versions are considered. We find tuning is critical for flexible models, while its effect is less important with rigid models. Models like tuned LightGBM and tuned random forests align closely with the benchmark, while weakly tuned neural networks continue to exhibit bias. The default random forest implementation yields an estimate of -55.409 (83.877), driven by very unstable propensity weights, and is thus not pictured in the figure. In contrast to the untrimmed comparison in Figure \ref{['fig:dr_comparison_untrimmed']}, we see trimming substantively reduces estimator variability and brings many models closer to the experimental benchmark, partially mitigating the gains from tuning. These results altogether reinforce that trimming and tuning are complementary tools for improving the reliability of observational causal estimation.
• Figure 5: Model Uncertainty: ATE Estimates