Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Learning with Abstention

Ayush Sawarni, Jikai Jin, Justin Whitehouse, Vasilis Syrgkanis

TL;DR

Policy learning with abstention addresses high-stakes decision settings by allowing learned policies to output a deferral symbol $\ast$ and receive a bonus $p$ when abstaining. The authors develop a two-stage abstention learner that first identifies near-optimal policies and then builds abstention rules from their disagreements, achieving fast $O\big(\frac{1}{n}\big)$ regret when propensities are known and extending to unknown propensities via a doubly robust estimator. They establish fast rates under margin-like conditions and connect abstention to distributional robustness and safe policy improvement, with practical algorithms for both known and unknown propensity settings. The framework yields improved guarantees and safer deployment in domains such as personalized medicine and policy optimization under shifts, while remaining compatible with standard policy-learning objectives and estimators.

Abstract

Policy learning algorithms are widely used in areas such as personalized medicine and advertising to develop individualized treatment regimes. However, most methods force a decision even when predictions are uncertain, which is risky in high-stakes settings. We study policy learning with abstention, where a policy may defer to a safe default or an expert. When a policy abstains, it receives a small additive reward on top of the value of a random guess. We propose a two-stage learner that first identifies a set of near-optimal policies and then constructs an abstention rule from their disagreements. We establish fast O(1/n)-type regret guarantees when propensities are known, and extend these guarantees to the unknown-propensity case via a doubly robust (DR) objective. We further show that abstention is a versatile tool with direct applications to other core problems in policy learning: it yields improved guarantees under margin conditions without the common realizability assumption, connects to distributionally robust policy learning by hedging against small data shifts, and supports safe policy improvement by ensuring improvement over a baseline policy with high probability.

Policy Learning with Abstention

TL;DR

Policy learning with abstention addresses high-stakes decision settings by allowing learned policies to output a deferral symbol and receive a bonus when abstaining. The authors develop a two-stage abstention learner that first identifies near-optimal policies and then builds abstention rules from their disagreements, achieving fast regret when propensities are known and extending to unknown propensities via a doubly robust estimator. They establish fast rates under margin-like conditions and connect abstention to distributional robustness and safe policy improvement, with practical algorithms for both known and unknown propensity settings. The framework yields improved guarantees and safer deployment in domains such as personalized medicine and policy optimization under shifts, while remaining compatible with standard policy-learning objectives and estimators.

Abstract

Policy learning algorithms are widely used in areas such as personalized medicine and advertising to develop individualized treatment regimes. However, most methods force a decision even when predictions are uncertain, which is risky in high-stakes settings. We study policy learning with abstention, where a policy may defer to a safe default or an expert. When a policy abstains, it receives a small additive reward on top of the value of a random guess. We propose a two-stage learner that first identifies a set of near-optimal policies and then constructs an abstention rule from their disagreements. We establish fast O(1/n)-type regret guarantees when propensities are known, and extend these guarantees to the unknown-propensity case via a doubly robust (DR) objective. We further show that abstention is a versatile tool with direct applications to other core problems in policy learning: it yields improved guarantees under margin conditions without the common realizability assumption, connects to distributionally robust policy learning by hedging against small data shifts, and supports safe policy improvement by ensuring improvement over a baseline policy with high probability.
Paper Structure (32 sections, 22 theorems, 149 equations, 4 figures, 4 algorithms)

This paper contains 32 sections, 22 theorems, 149 equations, 4 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related work on abstention.
  3. Policy Learning.
  4. Notation and Preliminaries
  5. Policies.
  6. Policy value.
  7. Policy Learning with Abstention
  8. Known Propensities
  9. Unknown Propensities: Doubly Robust Learner
  10. Applications
  11. Fast Learning Rates Without Standard Margin Assumptions
  12. Safe Policy Improvement
  13. Safe Policy Improvement Experiments
  14. Robustness to Distribution Shifts
  15. Conclusion
  16. ...and 17 more sections

Key Result

Theorem 3.1

Fix $p>0$ and $\delta\in(0,1)$. Under Assumption assump:data and Assumption assump:vc and the construction in Algorithm algo:abs, with probability at least $1-\delta$,

Figures (4)

  • Figure 1: Safe Policy Improvement — performance across varying noise variance. We compare Algorithm 2 (ours), HCPI (two variants), and Safe EWM.
  • Figure 2: Safe Policy Improvement — performance across varying gap between baseline and optimal policy value. We compare Algorithm \ref{['algo:safe_policy_abstention']} (ours), HCPI (two variants), and Safe EWM.
  • Figure 3: Mean value difference under abstention.
  • Figure 4: Winning rate of Algorithm \ref{['algo:abs']} compared with EWM.

Theorems & Definitions (42)

  • Theorem 3.1
  • Remark 3.2
  • Proposition 3.3: ERM benchmark at $p=0$
  • Theorem 3.4
  • Remark 3.5
  • Theorem 4.1
  • Theorem 4.2
  • Proposition 4.3
  • Lemma A.1
  • Lemma A.2
  • ...and 32 more