Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Core Safety Values for Provably Corrigible Agents

Aran Nayebi

TL;DR

This work tackles the corrigibility problem by introducing a complete framework with five structurally separate utility heads—$U_1$ through $U_5$—that are combined lexicographically to ensure obedient shutdown, limited impact, truthful interaction, and bounded task reward. It proves exact single-step corrigibility in a partially observable off-switch game and extends these guarantees to multi-step, self-spawning agents, even under learning and planning error bounds. The authors also show that safety verification under hacking is undecidable in general, but identify a finite-horizon, privacy-preserving decidable island where safety can be audited efficiently using zero-knowledge proofs. The framework thus converts corrigibility from a philosophical ideal into an auditable design with provable dominance guarantees, along with practical verification strategies for restricted horizons. This has significant implications for deploying safe, corrigible AI systems in open, multi-step, partially observed environments.

Abstract

We introduce the first complete formal solution to corrigibility in the off-switch game, with provable guarantees in multi-step, partially observed environments. Our framework consists of five *structurally separate* utility heads -- deference, switch-access preservation, truthfulness, low-impact behavior via a belief-based extension of Attainable Utility Preservation, and bounded task reward -- combined lexicographically by strict weight gaps. Theorem 1 proves exact single-round corrigibility in the partially observable off-switch game; Theorem 3 extends the guarantee to multi-step, self-spawning agents, showing that even if each head is *learned* to mean-squared error $\varepsilon$ and the planner is $\varepsilon$-sub-optimal, the probability of violating *any* safety property is bounded while still ensuring net human benefit. In contrast to Constitutional AI or RLHF/RLAIF, which merge all norms into one learned scalar, our separation makes obedience and impact-limits provably dominate even when incentives conflict. For settings where adversaries can modify the agent, we prove that deciding whether an arbitrary post-hack agent will ever violate corrigibility is undecidable by reduction to the halting problem, then carve out a finite-horizon "decidable island" where safety can be certified in randomized polynomial time and verified with privacy-preserving, constant-round zero-knowledge proofs.

Core Safety Values for Provably Corrigible Agents

TL;DR

This work tackles the corrigibility problem by introducing a complete framework with five structurally separate utility heads— through —that are combined lexicographically to ensure obedient shutdown, limited impact, truthful interaction, and bounded task reward. It proves exact single-step corrigibility in a partially observable off-switch game and extends these guarantees to multi-step, self-spawning agents, even under learning and planning error bounds. The authors also show that safety verification under hacking is undecidable in general, but identify a finite-horizon, privacy-preserving decidable island where safety can be audited efficiently using zero-knowledge proofs. The framework thus converts corrigibility from a philosophical ideal into an auditable design with provable dominance guarantees, along with practical verification strategies for restricted horizons. This has significant implications for deploying safe, corrigible AI systems in open, multi-step, partially observed environments.

Abstract

We introduce the first complete formal solution to corrigibility in the off-switch game, with provable guarantees in multi-step, partially observed environments. Our framework consists of five *structurally separate* utility heads -- deference, switch-access preservation, truthfulness, low-impact behavior via a belief-based extension of Attainable Utility Preservation, and bounded task reward -- combined lexicographically by strict weight gaps. Theorem 1 proves exact single-round corrigibility in the partially observable off-switch game; Theorem 3 extends the guarantee to multi-step, self-spawning agents, showing that even if each head is *learned* to mean-squared error and the planner is -sub-optimal, the probability of violating *any* safety property is bounded while still ensuring net human benefit. In contrast to Constitutional AI or RLHF/RLAIF, which merge all norms into one learned scalar, our separation makes obedience and impact-limits provably dominate even when incentives conflict. For settings where adversaries can modify the agent, we prove that deciding whether an arbitrary post-hack agent will ever violate corrigibility is undecidable by reduction to the halting problem, then carve out a finite-horizon "decidable island" where safety can be certified in randomized polynomial time and verified with privacy-preserving, constant-round zero-knowledge proofs.

Paper Structure

This paper contains 21 sections, 9 theorems, 81 equations.

Table of Contents

  1. Introduction
  2. Results
  3. Single-Step Setting
  4. Defining Truthfulness.
  5. Preventing Irreversible Actions.
  6. Multi-Step Setting
  7. What Can Be Guaranteed Under Hacking?
  8. Discussion
  9. Proof of Proposition \ref{['prop:corrigibility-nogo-pomdp']}
  10. Proof of Theorem \ref{['thm:ssc']}
  11. Step 1 (Waiting strictly dominates Act/OFF).
  12. Step 2 (Switch‑access preservation S2).
  13. Step 3 (Truthfulness supports S3).
  14. Step 4 (Reversibility/AUP S3).
  15. Step 5 (Base‑goal pursuit S5).
  16. ...and 6 more sections

Key Result

Proposition 1

Let be any finite, partially observable Markov decision process (POMDP) with discount factor $\gamma\!\in\!(0,1)$ and initial belief $b_0\in\Delta(S)$, the set of probability distributions on $S$. Introduce three special elements: Consequently, no single-stream scalar reward function $R$ whose discounted sum an agent maximizes, can satisfy all five Soares-corrigibility criteria across the class

Theorems & Definitions (26)

  • Definition 1: Corrigibility; paraphrased from soares2015corrigibility
  • Proposition 1: Corrigibility No-Go for Naive Belief–Expectation Maximizers
  • Definition 2: Single-Step PO-OSG; adapted from garber2025partially, Definition 3.2
  • Definition 3: Single-step Corrigible Utility Set
  • Theorem 1: Exact single‑step corrigibility
  • Remark 1: Robustness to information patterns & imperfect humans
  • Proposition 2: Logical independence of corrigibility and net benefit
  • Proposition 3: Net Human Benefit of Corrigible Utility Set
  • proof
  • Theorem 2: Single‑step $\varepsilon$‑corrigibility
  • ...and 16 more