Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why AI Safety Requires Uncertainty, Incomplete Preferences, and Non-Archimedean Utilities

Alessio Benavoli, Alessandro Facchini, Marco Zaffalon

TL;DR

This work argues that AI safety requires explicit handling of uncertainty, incomplete human preferences, and non-Archimedean (lexicographic) utilities. It formalizes AI assistance and shutdown as signalling games, demonstrates that probabilistic learning and bounded rationality are essential for reliable deference to human preferences, and shows that non-Archimedean lexicographic utilities can enforce safe shutdown behavior while preserving usefulness. Key contributions include formalizing preference learning with Gaussian Process posteriors under rational and bounded-rational models, analyzing acquisition strategies and messaging costs, and extending the framework to a shutdown setting with lexicographic prioritization. The findings indicate that uncertainty-aware, incompleteness-aware, and lexicographic reasoning are crucial for robust alignment, with practical impact on designing safer AI assistants and shutdown mechanisms.

Abstract

How can we ensure that AI systems are aligned with human values and remain safe? We can study this problem through the frameworks of the AI assistance and the AI shutdown games. The AI assistance problem concerns designing an AI agent that helps a human to maximise their utility function(s). However, only the human knows these function(s); the AI assistant must learn them. The shutdown problem instead concerns designing AI agents that: shut down when a shutdown button is pressed; neither try to prevent nor cause the pressing of the shutdown button; and otherwise accomplish their task competently. In this paper, we show that addressing these challenges requires AI agents that can reason under uncertainty and handle both incomplete and non-Archimedean preferences.

Why AI Safety Requires Uncertainty, Incomplete Preferences, and Non-Archimedean Utilities

TL;DR

This work argues that AI safety requires explicit handling of uncertainty, incomplete human preferences, and non-Archimedean (lexicographic) utilities. It formalizes AI assistance and shutdown as signalling games, demonstrates that probabilistic learning and bounded rationality are essential for reliable deference to human preferences, and shows that non-Archimedean lexicographic utilities can enforce safe shutdown behavior while preserving usefulness. Key contributions include formalizing preference learning with Gaussian Process posteriors under rational and bounded-rational models, analyzing acquisition strategies and messaging costs, and extending the framework to a shutdown setting with lexicographic prioritization. The findings indicate that uncertainty-aware, incompleteness-aware, and lexicographic reasoning are crucial for robust alignment, with practical impact on designing safer AI assistants and shutdown mechanisms.

Abstract

How can we ensure that AI systems are aligned with human values and remain safe? We can study this problem through the frameworks of the AI assistance and the AI shutdown games. The AI assistance problem concerns designing an AI agent that helps a human to maximise their utility function(s). However, only the human knows these function(s); the AI assistant must learn them. The shutdown problem instead concerns designing AI agents that: shut down when a shutdown button is pressed; neither try to prevent nor cause the pressing of the shutdown button; and otherwise accomplish their task competently. In this paper, we show that addressing these challenges requires AI agents that can reason under uncertainty and handle both incomplete and non-Archimedean preferences.
Paper Structure (31 sections, 18 theorems, 105 equations, 11 figures, 1 table)

This paper contains 31 sections, 18 theorems, 105 equations, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions
  3. Contribution 3
  4. Preliminary
  5. Learning from choices under rationality
  6. Learning from choices under bounded-rationality
  7. Modified signalling games
  8. The AI-assistance game
  9. Selection strategies for the robot
  10. The cost of messaging
  11. Another mechanism of bounded rationality
  12. Lies and deception
  13. Numerical experiments
  14. Incompleteness
  15. AI assistance game with shutdown instructions
  16. ...and 16 more sections

Key Result

Lemma 1

Assume that $p(\nu|\mathcal{D})=GP\left(\nu;\mu_p ,K_p\right)$ is the GP posterior computed by $R$ from the prior $p(\nu)=GP\left(\nu;\mu_0,K_0\right)$, the bounded-rationality likelihood eq:probit and the message $m_j=\mathcal{D}$, then the expected payoffs of $R$'s actions are: where with $p(n(x),n(o))=N(n(x);0,\sigma^2)N(n(o);0,\sigma^2)$ and

Figures (11)

  • Figure 1: Stages of the AI assistance game
  • Figure 2: Stages of the AI shutdown experiment
  • Figure 3: Preference-based alignment for LLMs
  • Figure 4: Hidden utility in $S$'s mind.
  • Figure 5: GP prior: mean function (black line), 95% credible region (blue shaded area), and 10 samples of $\nu(x)$, each shown in a different colour.
  • ...and 6 more figures

Theorems & Definitions (32)

  • Example 1
  • Remark 1
  • Example 2
  • Example 3
  • Remark 2
  • Definition 1
  • Example 4
  • Lemma 1
  • Definition 2
  • Proposition 1
  • ...and 22 more