Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning When Not to Learn: Risk-Sensitive Abstention in Bandits with Unbounded Rewards

Sarah Liaw, Benjamin Plaut

TL;DR

This paper addresses learning under irrecoverable costs by introducing a mentor-free abstention mechanism in a two-action contextual bandit with unbounded rewards. It proposes a cautious algorithm that restricts learning to a Lipschitz-continuous, non-OOD region around the origin, certifying bins as unsafe when evidence suggests potential harm, and abstaining elsewhere. The main theoretical result shows a sublinear regret bound of order $\mathcal{O}\big((L+\sigma^2) T^{(n+1)/(n+2)} (\ln T)^{n+1} + T \bar{\nu}(\ln T)\big)$, highlighting a tail term that depends on the input distribution's OOD probability. The work provides a principled approach for deploying learning agents safely in high-stakes environments by trading off exploration with risk there-by enabling practical, mentor-free safety.

Abstract

In high-stakes AI applications, even a single action can cause irreparable damage. However, nearly all of sequential decision-making theory assumes that all errors are recoverable (e.g., by bounding rewards). Standard bandit algorithms that explore aggressively may cause irreparable damage when this assumption fails. Some prior work avoids irreparable errors by asking for help from a mentor, but a mentor may not always be available. In this work, we formalize a model of learning with unbounded rewards without a mentor as a two-action contextual bandit with an abstain option: at each round the agent observes an input and chooses either to abstain (always 0 reward) or to commit (execute a preexisting task policy). Committing yields rewards that are upper-bounded but can be arbitrarily negative, and the commit reward is assumed Lipschitz in the input. We propose a caution-based algorithm that learns when not to learn: it chooses a trusted region and commits only where the available evidence does not already certify harm. Under these conditions and i.i.d. inputs, we establish sublinear regret guarantees, theoretically demonstrating the effectiveness of cautious exploration for deploying learning agents safely in high-stakes environments.

Learning When Not to Learn: Risk-Sensitive Abstention in Bandits with Unbounded Rewards

TL;DR

This paper addresses learning under irrecoverable costs by introducing a mentor-free abstention mechanism in a two-action contextual bandit with unbounded rewards. It proposes a cautious algorithm that restricts learning to a Lipschitz-continuous, non-OOD region around the origin, certifying bins as unsafe when evidence suggests potential harm, and abstaining elsewhere. The main theoretical result shows a sublinear regret bound of order , highlighting a tail term that depends on the input distribution's OOD probability. The work provides a principled approach for deploying learning agents safely in high-stakes environments by trading off exploration with risk there-by enabling practical, mentor-free safety.

Abstract

In high-stakes AI applications, even a single action can cause irreparable damage. However, nearly all of sequential decision-making theory assumes that all errors are recoverable (e.g., by bounding rewards). Standard bandit algorithms that explore aggressively may cause irreparable damage when this assumption fails. Some prior work avoids irreparable errors by asking for help from a mentor, but a mentor may not always be available. In this work, we formalize a model of learning with unbounded rewards without a mentor as a two-action contextual bandit with an abstain option: at each round the agent observes an input and chooses either to abstain (always 0 reward) or to commit (execute a preexisting task policy). Committing yields rewards that are upper-bounded but can be arbitrarily negative, and the commit reward is assumed Lipschitz in the input. We propose a caution-based algorithm that learns when not to learn: it chooses a trusted region and commits only where the available evidence does not already certify harm. Under these conditions and i.i.d. inputs, we establish sublinear regret guarantees, theoretically demonstrating the effectiveness of cautious exploration for deploying learning agents safely in high-stakes environments.
Paper Structure (12 sections, 19 theorems, 38 equations, 1 figure, 1 algorithm)

This paper contains 12 sections, 19 theorems, 38 equations, 1 figure, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Sequential decision-making when all errors are recoverable
  4. Safe exploration
  5. Other related work
  6. PRELIMINARIES
  7. THE VIRTUES AND LIMITS OF CAUTION
  8. ALGORITHM AND MAIN RESULT
  9. PROOF SKETCH
  10. CONCLUSION
  11. PROOF OF MAIN RESULT
  12. Proof notation

Key Result

Theorem 4.1

Let $\nu$ be any distribution over $\mathcal{X}$ such that $\mathop{\mathrm{\mathbb{E}}}\limits_{x\sim \nu}[\|x\|] = \infty$ and assume $x_1,\dots,x_T \sim \nu$ i.i.d. Then there exists a reward function $r$ such that any algorithm which always commits on the first time step satisfies $\mathop{\math

Figures (1)

  • Figure 1: Trusted region of radius $m(T)$ around the origin, partitioned into bins of side $w(T)$. Any square intersecting the ball is shown fully green (bin $\in\mathcal{B}$). Certified negative bins are shown hatched red. The agent abstains outside the ball.

Theorems & Definitions (31)

  • Definition 3.1: $\sigma$-subgaussian
  • Theorem 4.1: The need for caution
  • proof
  • Theorem 4.2: The limits of caution
  • proof
  • Definition 5.1: Radial survival function
  • Theorem 5.2
  • Lemma 6.0: Per-bin concentration
  • Lemma 6.0: Uniform concentration bound
  • Lemma 6.0: Samples for negative certification
  • ...and 21 more