Comparing Targeting Strategies for Maximizing Social Welfare with Limited Resources

TL;DR

The paper investigates how to optimally allocate scarce interventions by comparing risk-based targeting (predicting baseline risk) with targeting based on estimated heterogeneous treatment effects (TEs) across five real-world RCTs. Using a doubly-robust framework to estimate CATEs and a kernel-based approach to relate TE to baseline risk, the authors show that TE-based targeting yields substantially higher welfare when TE estimation is reliable, even under confounding and inequality-averse preferences. However, practical data limitations often hinder learning accurate TE mappings, explaining the continued popularity of risk-based targeting. The work emphasizes that richer data and better causal modeling can unlock the potential of causal targeting, suggesting a shift toward investing in data collection and advanced estimation while acknowledging biases in observational data.

Abstract

Machine learning is increasingly used to select which individuals receive limited-resource interventions in domains such as human services, education, development, and more. However, it is often not apparent what the right quantity is for models to predict. Policymakers rarely have access to data from a randomized controlled trial (RCT) that would enable accurate estimates of which individuals would benefit more from the intervention, while observational data creates a substantial risk of bias in treatment effect estimates. Practitioners instead commonly use a technique termed ``risk-based targeting" where the model is just used to predict each individual's status quo outcome (an easier, non-causal task). Those with higher predicted risk are offered treatment. There is currently almost no empirical evidence to inform which choices lead to the most effective machine learning-informed targeting strategies in social domains. In this work, we use data from 5 real-world RCTs in a variety of domains to empirically assess such choices. We find that when treatment effects can be estimated with high accuracy (which we simulate by allowing the model to partially observe outcomes in advance), treatment effect based targeting substantially outperforms risk-based targeting, even when treatment effect estimates are biased. Moreover, these results hold even when the policymaker has strong normative preferences for assisting higher-risk individuals. However, the features and data actually available in most RCTs we examine do not suffice for accurate estimates of heterogeneous treatment effects. Our results suggest treatment effect targeting has significant potential benefits, but realizing these benefits requires improvements to data collection and model training beyond what is currently common in practice.

Figures (18)

• Figure 1: Observing treatment effect heterogeneity across different settings by plotting treatment effect against baseline risk for each of our 5 datasets. We observe a unique trend for each dataset, indicating a lack of a consistent well-defined relation between the two quantities
• Figure 2: Comparison of risk-based targeting to biased treatment effect-based targeting by plotting the benefit offered by each policy against the amount of data systematically removed from the RCT to introduce confounding
• Figure 3: Comparison of risk-based targeting to biased treatment effect-based targeting by plotting the benefit offered by each policy against the amount of data systematically removed from the RCT to introduce confounding but using observed pseudo-outcome data
• Figure 4: Comparing risk-based targeting to biased treatment effect-based targeting on weighted utilitarian welfare and Nash social welfare for the STAR RCT.
• Figure 5: Comparing risk-based targeting to biased treatment effect-based targeting on weighted utilitarian welfare and Nash social welfare for the Ultra Poor RCT.