The Partially Observable Off-Switch Game

TL;DR

The paper addresses AI shutdown incentives under asymmetric information by introducing the Partially Observable Off-Switch Game (PO-OSG), a dynamic Bayesian, common-payoff model with partial observability. It shows that under PO-OSGs, an AI can have incentives to avoid shutdown even when the human is perfectly rational, and analyzes how information structure and both bounded and unbounded communication shape deference through multiple results, including the nonmonotonic effects of information gain and the presence of implicit communication. A range of contributions are provided: (i) examples where A avoids shutdown in optimal play, (ii) formal results on informativeness via coordinated garblings, (iii) analyses of PO-OSG-C with cheap talk, (iv) complexity results for solving PO-OSGs with and without human awareness, and (v) discussion of A-unaware human policies and their impact. The findings highlight the delicate tradeoffs between payoff maximization and maintaining human-AI deference, offering guidance for designing safer, corrigible AIs in settings with asymmetric information.

Abstract

A wide variety of goals could cause an AI to disable its off switch because "you can't fetch the coffee if you're dead" (Russell 2019). Prior theoretical work on this shutdown problem assumes that humans know everything that AIs do. In practice, however, humans have only limited information. Moreover, in many of the settings where the shutdown problem is most concerning, AIs might have vast amounts of private information. To capture these differences in knowledge, we introduce the Partially Observable Off-Switch Game (PO-OSG), a game-theoretic model of the shutdown problem with asymmetric information. Unlike when the human has full observability, we find that in optimal play, even AI agents assisting perfectly rational humans sometimes avoid shutdown. As expected, increasing the amount of communication or information available always increases (or leaves unchanged) the agents' expected common payoff. But counterintuitively, introducing bounded communication can make the AI defer to the human less in optimal play even though communication mitigates information asymmetry. In particular, communication sometimes enables new optimal behavior requiring strategic AI deference to achieve outcomes that were previously inaccessible. Thus, designing safe artificial agents in the presence of asymmetric information requires careful consideration of the tradeoffs between maximizing payoffs (potentially myopically) and maintaining AIs' incentives to defer to humans.

Figures (6)

• Figure 1: The basic setup of a Partially Observable Off-Switch Game (PO-OSG). A state is selected randomly and the human $\mathbf{H}$ and AI assistant $\mathbf{A}$ receive (possibly dependent) observations. Then, each agent acts. $\mathbf{A}$ may wait ($w(a)$), disable the off-switch and act ($a$), or shut down (OFF). If $\mathbf{A}$ waits, $\mathbf{H}$ may let $\mathbf{A}$ act (ON) or turn $\mathbf{A}$ off (OFF). $\mathbf{A}$ and $\mathbf{H}$ share a common payoff $u_a(S)$ if the action goes through and $u_o(S)$ if not. \ref{['df:bosg']} formally defines PO-OSGs.
• Figure 2: This figure illustrates an intuition that we demonstrate does not hold. Although the AI in a PO-OSG has no incentive to defer when it knows everything the human knows, and has incentive to always defer when the human knows everything it knows, there are cases when making the human more informed or the AI less informed (i.e., moving to the right in the diagram above) can give the AI incentives to defer less. \ref{['fig:hum-informed-matrices1', 'fig:rob-informed']} depict examples of such cases.
• Figure 2.a: Expected payoff $=1$
• Figure 3.a: Expected payoff $=1$
• Figure 4.a: Expected payoff $=1$

Theorems & Definitions (64)

• Definition 3.1
• Definition 3.2
• Example 4.1: The File Deletion Game
• Definition 4.2
• Proposition 4.2
• Definition 4.3
• Definition 4.4
• Definition 4.5
• Theorem 4.6
• Definition 4.7