Consequences of Misaligned AI

TL;DR

The paper tackles the problem that AI reward proxies may incompletely capture human goals, creating a principal–agent misalignment. It introduces a formal model with a resource-constrained state space $S ⊆ R^L$, a constraint $C(s) ≤ 0$, and a proxy on a subset of attributes $J$ (with $J < L$), deriving necessary and sufficient conditions under which optimizing the incomplete proxy can render utility arbitrarily poor. It shows that without intervention, overoptimization is unavoidable; however, interactive reward learning and impact minimization can mitigate misalignment and yield improved outcomes, connecting to broader value-alignment strategies. The work provides a theoretical justification for dynamic, human-in-the-loop reward design and lays groundwork for practical mitigations in algorithmic systems such as content recommenders.

Abstract

AI systems often rely on two key components: a specified goal or reward function and an optimization algorithm to compute the optimal behavior for that goal. This approach is intended to provide value for a principal: the user on whose behalf the agent acts. The objectives given to these agents often refer to a partial specification of the principal's goals. We consider the cost of this incompleteness by analyzing a model of a principal and an agent in a resource constrained world where the $L$ attributes of the state correspond to different sources of utility for the principal. We assume that the reward function given to the agent only has support on $J < L$ attributes. The contributions of our paper are as follows: 1) we propose a novel model of an incomplete principal-agent problem from artificial intelligence; 2) we provide necessary and sufficient conditions under which indefinitely optimizing for any incomplete proxy objective leads to arbitrarily low overall utility; and 3) we show how modifying the setup to allow reward functions that reference the full state or allowing the principal to update the proxy objective over time can lead to higher utility solutions. The results in this paper argue that we should view the design of reward functions as an interactive and dynamic process and identifies a theoretical scenario where some degree of interactivity is desirable.

Figures (2)

• Figure 1: Our model of the principal---agent problem in AI. Starting from an initial state $s^{(0)}$, the robot eventually outputs a mapping from time $t \in \mathbb{Z}^+$ to states. Left: The human gives the robot a single proxy utility function to optimize for all time. We prove that this paradigm reliably leads to the human actor losing utility, compared to the initial state. Right: An interactive solution, where the human changes the proxy utility function at regular time intervals depending on the current allocation of resources. We show that, at least in theory, this approach does produce value for the human, even under adversarial assumptions.
• Figure 2: An illustrative example of our model with $L=4$ and $J=2$. Left: Proxy utility and true utility eventually diverge as the agent overallocates resources from unreferenced attributes to the proxy variables. Right: The true utility generated by optimizing all pairs of proxy attributes. The utility generation is eventually negative in all cases because this example meets the conditions of Theorem 2.

Theorems & Definitions (16)

• Theorem 1
• proof
• Theorem 2
• proof
• Proposition 1
• proof
• Proposition 2
• proof
• Proposition 3
• proof