Improving Bias Correction Standards by Quantifying its Effects on Treatment Outcomes

TL;DR

This work addresses the problem of extreme variability in causal estimates from propensity score matching (PSM) pipelines in retrospective health studies. It introduces A2A, a novel end-to-end metric that evaluates bias correction by simulating artificial tasks with known outcomes, and combines it with Standardized Mean Difference (SMD) to more reliably select robust PSM pipelines. Implemented in the Popmatch Python package, the approach automates propensity-score estimation, matching, and validation, integrating methods from both Python and R and enabling large-scale, reproducible experiments. Across synthetic and real datasets, A2A reduces $ATE$ estimation errors by up to 50% and $ATE$ variability by up to 90%, with a Pareto-based fusion of $SMD$ and $A2A$ yielding robust, generalizable results. The authors argue for automated pipelines and benchmarks to standardize bias correction in health data, facilitating regulatory review and broader methodological transparency.

Abstract

With the growing access to administrative health databases, retrospective studies have become crucial evidence for medical treatments. Yet, non-randomized studies frequently face selection biases, requiring mitigation strategies. Propensity score matching (PSM) addresses these biases by selecting comparable populations, allowing for analysis without further methodological constraints. However, PSM has several drawbacks. Different matching methods can produce significantly different Average Treatment Effects (ATE) for the same task, even when meeting all validation criteria. To prevent cherry-picking the best method, public authorities must involve field experts and engage in extensive discussions with researchers. To address this issue, we introduce a novel metric, A2A, to reduce the number of valid matches. A2A constructs artificial matching tasks that mirror the original ones but with known outcomes, assessing each matching method's performance comprehensively from propensity estimation to ATE estimation. When combined with Standardized Mean Difference, A2A enhances the precision of model selection, resulting in a reduction of up to 50% in ATE estimation errors across synthetic tasks and up to 90% in predicted ATE variability across both synthetic and real-world datasets. To our knowledge, A2A is the first metric capable of evaluating outcome correction accuracy using covariates not involved in selection. Computing A2A requires solving hundreds of PSMs, we therefore automate all manual steps of the PSM pipeline. We integrate PSM methods from Python and R, our automated pipeline, a new metric, and reproducible experiments into popmatch, our new Python package, to enhance reproducibility and accessibility to bias correction methods.

Figures (3)

• Figure 1: Propensity score matching pipeline. Blue boxes indicate steps where the practitioner makes choices. Red boxes indicates steps ending with a validation. Backward arrows show points where the practitioner may revisit previous decisions.
• Figure 2: Example of propensity score histograms for control and treated population. The hatched area corresponds to the overlap between the two.
• Figure B.3: Values for SMD and A2A for the task NHANES. Methods above 0.1 in SMD are invalid. Bipartify LR and GLM form the Pareto optimum according to both metrics.