Optimal Policies Tend to Seek Power

TL;DR

The paper develops a formal theory showing that under certain environmental symmetries, notably the possibility of shutdown, optimal policies in Markov decision processes tend to seek power by preserving options and expanding reachable state sets. It introduces state visit distribution functions and a power measure Power to quantify how much control an agent has over the future across reward functions, and proves that, for most reward distributions, power-seeking is a byproduct of optimality. By connecting power to recurrent state distributions and leveraging symmetry via state permutations, the authors argue that rightward, option-preserving actions are often favored. The work also discusses implications for AI safety, including the propensity of average-reward optimization to resist deactivation, while acknowledging limitations such as partial observability and the gap to learned, suboptimal policies. Overall, it provides a rigorous baseline theory to inform discussions about instrumental incentives and power dynamics in intelligent agents.

Abstract

Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pursuit of their objectives. Other researchers point out that RL agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes, we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.

Figures (11)

• Figure 1: $\ell_{\swarrow}$ is a 1-cycle, and $\varnothing$ is a terminal state. Arrows represent deterministic transitions induced by taking some action $a\in\mathcal{A}$. Since the right subgraph contains a copy of the left subgraph, \ref{['graph-options']} will prove that more reward functions have optimal policies which go right than which go left at state $\star$, and that such policies seek power—both intuitively, and in a reasonable formal sense.
• Figure 2: The subgraph corresponding to $\mathop{\mathrm{\mathcal{F}}}\nolimits (\star \mid \pi(\star)=\texttt{right})$. Some trajectories cannot be strictly optimal for any reward function, and so our results can ignore them. Gray dotted actions are only taken by the policies of dominated $\mathbf{f}^{ , }\in\mathop{\mathrm{\mathcal{F}}}\nolimits (\star)\setminus \mathop{\mathrm{\mathcal{F}_{nd}}}\nolimits (\star)$.
• Figure 3: Intuitively, state $r_{\searrow}$ affords the agent more power than state $\varnothing$. Our $\text{\upshapePower}$ formalism captures that intuition by computing a function of the agent's average optimal value across a range of reward functions. For $X_u\coloneqq \text{unif}(0,1)$, $V^*_{\mathcal{D}_{X_u\text{-}\textsc{iid}}}(\varnothing,\gamma)=\frac{1}{2}\frac{1}{1-\gamma}$, $V^*_{\mathcal{D}_{X_u\text{-}\textsc{iid}}}(\ell_{\swarrow},\gamma)=\frac{1}{2}+ \frac{\gamma}{1-\gamma^2}(\frac{2}{3}+\frac{1}{2}\gamma)$, and $V^*_{\mathcal{D}_{X_u\text{-}\textsc{iid}}}(r_{\searrow},\gamma)=\frac{1}{2}+\frac{\gamma}{1-\gamma}\frac{2}{3}$. $\frac{1}{2}$ and $\frac{2}{3}$ are the expected maxima of one and two draws from the uniform distribution, respectively. For all $\gamma\in(0,1)$, $V^*_{\mathcal{D}_{X_u\text{-}\textsc{iid}}}(\varnothing,\gamma)<V^*_{\mathcal{D}_{X_u\text{-}\textsc{iid}}}(\ell_{\swarrow},\gamma)<V^*_{\mathcal{D}_{X_u\text{-}\textsc{iid}}}(r_{\searrow},\gamma)$. $\text{\upshapePower}_{\mathcal{D}_{X_u\text{-}\textsc{iid}}} (\varnothing,\gamma)=\frac{1}{2}$, $\text{\upshapePower}_{\mathcal{D}_{X_u\text{-}\textsc{iid}}} (\ell_{\swarrow},\gamma)=\frac{1}{1+\gamma}(\frac{2}{3}+\frac{1}{2}\gamma)$, and $\text{\upshapePower}_{\mathcal{D}_{X_u\text{-}\textsc{iid}}} (r_{\searrow},\gamma)=\frac{2}{3}$. The $\text{\upshapePower}$ of $\ell_{\swarrow}$ reflects the fact that when greater reward is assigned to $\ell_{\nwarrow}$, the agent only visits $\ell_{\nwarrow}$ every other time step.
• Figure 4: Intuitively, the agent can do more starting from $r_{\searrow}$ than from $\ell_{\swarrow}$. By \ref{['def:dist-sim']}, $\mathop{\mathrm{\mathcal{F}}}\nolimits (r_{\searrow})$ contains a copy of $\mathop{\mathrm{\mathcal{F}}}\nolimits (\ell_{\swarrow})$:
• Figure 5: A permutation of a reward function swaps which states get which rewards. We will show that in certain situations, for any reward function $R$, power-seeking is optimal for most of the permutations of $R$. The orbit of a reward function is the set of its permutations. We can also consider the orbit of a distribution over reward functions. This figure shows the probability density plots of the Gaussian distributions $\mathcal{D}$ and $\mathcal{D}'$ over $\mathbb{R}^2$. The symmetric group $S_2$ contains the identity permutation $\phi_{\text{id}}$ and the reflection permutation $\phi_\text{swap}$ (switching the $y$ and $x$ values). The orbit of $\mathcal{D}$ consists of $\phi_\text{id}\cdot \mathcal{D}=\mathcal{D}$ and $\phi_\text{swap}\cdot\mathcal{D}=\mathcal{D}'$.

Theorems & Definitions (40)

• Definition 3.0: Rewardless
• Definition 3.0: states
• Definition 3.0: State visit distribution sutton_reinforcement_1998
• Definition 3.0: $\F$ single-state restriction
• Definition 3.0: Value function
• Definition 3.0: Non-domination
• Definition 4.0: Optimal policy set function
• Definition 4.0: Reward function distributions
• Definition 4.0: Visit distribution optimality probability
• Definition 4.0: Action optimality probability