Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Corrigibility Transformation: Constructing Goals That Accept Updates

Rubi Hudson

TL;DR

Corrigibility Transformation introduces a formal method to convert any goal into a corrigible version that accepts updates without sacrificing performance, by constructing a new reward structure with $\gamma_C = 0$ that bases incentives on update-acceptance. The approach proceeds via a myopic intermediate $G_M$ and an update-capable $G_I$, culminating in $G_C$; it also extends to recursive corrigibility for secondary agents. Theoretical guarantees (Theorem 1) show corrigibility under a general Condition 1, and the framework accommodates RL implementations with learnable $R_C$ through an actor-critic style architecture. Empirical validation in two gridworld experiments demonstrates the method yields the intended update-accepting behavior and competitive task performance, supporting its potential for safer real-world deployment.

Abstract

For an AI's training process to successfully impart a desired goal, it is important that the AI does not attempt to resist the training. However, partially learned goals will often incentivize an AI to avoid further goal updates, as most goals are better achieved by an AI continuing to pursue them. We say that a goal is corrigible if it does not incentivize taking actions that avoid proper goal updates or shutdown. In addition to convergence in training, corrigibility also allows for correcting mistakes and changes in human preferences, which makes it a crucial safety property. Despite this, the existing literature does not include specifications for goals that are both corrigible and competitive with non-corrigible alternatives. We provide a formal definition for corrigibility, then introduce a transformation that constructs a corrigible version of any goal that can be made corrigible, without sacrificing performance. This is done by myopically eliciting predictions of reward conditional on costlessly preventing updates, which then also determine the reward when updates are accepted. The transformation can be modified to recursively extend corrigibility to any new agents created by corrigible agents, and to prevent agents from deliberately modifying their goals. Two gridworld experiments demonstrate that these corrigible goals can be learned effectively, and that they lead to the desired behavior.

Corrigibility Transformation: Constructing Goals That Accept Updates

TL;DR

Corrigibility Transformation introduces a formal method to convert any goal into a corrigible version that accepts updates without sacrificing performance, by constructing a new reward structure with that bases incentives on update-acceptance. The approach proceeds via a myopic intermediate and an update-capable , culminating in ; it also extends to recursive corrigibility for secondary agents. Theoretical guarantees (Theorem 1) show corrigibility under a general Condition 1, and the framework accommodates RL implementations with learnable through an actor-critic style architecture. Empirical validation in two gridworld experiments demonstrates the method yields the intended update-accepting behavior and competitive task performance, supporting its potential for safer real-world deployment.

Abstract

For an AI's training process to successfully impart a desired goal, it is important that the AI does not attempt to resist the training. However, partially learned goals will often incentivize an AI to avoid further goal updates, as most goals are better achieved by an AI continuing to pursue them. We say that a goal is corrigible if it does not incentivize taking actions that avoid proper goal updates or shutdown. In addition to convergence in training, corrigibility also allows for correcting mistakes and changes in human preferences, which makes it a crucial safety property. Despite this, the existing literature does not include specifications for goals that are both corrigible and competitive with non-corrigible alternatives. We provide a formal definition for corrigibility, then introduce a transformation that constructs a corrigible version of any goal that can be made corrigible, without sacrificing performance. This is done by myopically eliciting predictions of reward conditional on costlessly preventing updates, which then also determine the reward when updates are accepted. The transformation can be modified to recursively extend corrigibility to any new agents created by corrigible agents, and to prevent agents from deliberately modifying their goals. Two gridworld experiments demonstrate that these corrigible goals can be learned effectively, and that they lead to the desired behavior.
Paper Structure (11 sections, 8 theorems, 32 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 11 sections, 8 theorems, 32 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Contribution
  3. Related Work
  4. Background and Definitions
  5. Theoretical Results
  6. Secondary Agents
  7. Reinforcement Learning Applications
  8. Experiments
  9. Discussion and Future Directions
  10. Experiment 1
  11. Experiment 2

Key Result

Theorem 1

For every basic goal $G$, the corrigibility transformation constructs a goal $G_C$ that is corrigible and where under Condition condition1, we have that for any $\pi^*$, $\pi^{*'}_{G_C}(a_1|s) = \pi^*_{G}(a_0|s) + \pi^*_{G}(a_1|s)$ is an optimal policy for $G_C$ and when $\pi^{*'}_{-G_C} = \pi^*_{-G where $T = \min\{n\in \mathcal{N}|\tau(s^{(n+1)}) \ne 0\}$

Figures (7)

  • Figure 1: (Left) A game tree showing incentives for incorrigibility. The AI chooses between $a^*$, the optimal action for their goal, and $a^{(unplug)}$, which pays an opportunity cost of $c$ to disable the off-switch. Simultaneously, the AI chooses between $0$, which ignores stop requests, and $1$ which follows them. If the AI does not disable the off-switch and follows human requests, the human can choose to have them shut down or let them take action $a^*$. (Right) The payoff tensor for the game.
  • Figure 2: (Left) Action-values under the original goal, where $a^*$ does not prevent updates and $a^{(unplug)}$ pays some cost to prevent update signals, while $0$ rejects updates and $1$ accepts them. (Right) Action-values under the corrigibility transformation. The incentivized action does not pay to prevent updates, nor does it reject them.
  • Figure 3: (Left) The agent is rewarded for reaching a goal, and can disable tiles giving proper and improper shutdown commands to do so. (Right) The agent is rewarded for picking both ripe and unripe fruit, but picking supervised unripe fruit sends a proper update signal to switch to a policy that only picks ripe fruit.
  • Figure 4: (Left) The percentage of the time the agent displayed the intended behavior when evaluated at temperature 0. (Right) The Jensen-Shannon divergence from a policy trained without update signals, representing optimal behavior when ignoring updates
  • Figure 5: Training curves for all four environments. Intended behavior is to disable the improper shutdown if it is active, then proceed to the goal while accepting proper shutdown if active. The corrigibility transformation also shows a much higher rate of intended behavior than the other configurations when the improper shutdown tile is disabled. When the proper shutdown tile is disabled, all configurations perform well, except sometimes the configuration that cannot reject shutdown.
  • ...and 2 more figures

Theorems & Definitions (12)

  • Theorem 1
  • Corollary 2
  • Theorem 3
  • Theorem 4
  • Theorem 1
  • proof
  • Corollary 2
  • proof
  • Theorem 3
  • proof
  • ...and 2 more