SNPL: Simultaneous Policy Learning and Evaluation for Safe Multi-Objective Policy Improvement
Brian Cho, Ana-Roxana Pop, Ariel Evnine, Nathan Kallus
TL;DR
SNPL addresses offline, multi-objective policy learning under low signal-to-noise by uniting stable, data-driven policy selection with joint post-selection confidence bounds. It provides finite-sample and asymptotic safety guarantees without data-splitting, using IPW and doubly robust estimators, cross-fitting, and an ε-stability framework to maintain valid inference after policy selection. The approach prunes candidate policies via a stability-based procedure and then selects the best policy whose post-correction lower bounds on guardrails and goals remain above zero, ensuring high-probability safety. Empirical validation on a real-world SMS personalization task demonstrates substantial gains in detection rates and goal improvements while maintaining strict safety guarantees, outperforming data-splitting and Bonferroni baselines, especially in large policy classes with low SNR.
Abstract
To design effective digital interventions, experimenters face the challenge of learning decision policies that balance multiple objectives using offline data. Often, they aim to develop policies that maximize goal outcomes, while ensuring there are no undesirable changes in guardrail outcomes. To provide credible recommendations, experimenters must not only identify policies that satisfy the desired changes in goal and guardrail outcomes, but also offer probabilistic guarantees about the changes these policies induce. In practice, however, policy classes are often large, and digital experiments tend to produce datasets with small effect sizes relative to noise. In this setting, standard approaches such as data splitting or multiple testing often result in unstable policy selection and/or insufficient statistical power. In this paper, we provide safe noisy policy learning (SNPL), a novel approach that leverages the concept of algorithmic stability to address these challenges. Our method enables policy learning while simultaneously providing high-confidence guarantees using the entire dataset, avoiding the need for data-splitting. We present finite-sample and asymptotic versions of our algorithm that ensure the recommended policy satisfies high-probability guarantees for avoiding guardrail regressions and/or achieving goal outcome improvements. We test both variants of our approach approach empirically on a real-world application of personalizing SMS delivery. Our results on real-world data suggest that our approach offers dramatic improvements in settings with large policy classes and low signal-to-noise across both finite-sample and asymptotic safety guarantees, offering up to 300\% improvements in detection rates and 150\% improvements in policy gains at significantly smaller sample sizes.