Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contracts

TL;DR

This paper proposes a framework where a principal guides an agent in a Markov Decision Process using a series of contracts, which specify payments by the principal based on observable outcomes of the agent's actions, and presents and analyze a meta-algorithm that iteratively optimizes the policies of the principal and agent.

Abstract

The increasing deployment of AI is shaping the future landscape of the internet, which is set to become an integrated ecosystem of AI agents. Orchestrating the interaction among AI agents necessitates decentralized, self-sustaining mechanisms that harmonize the tension between individual interests and social welfare. In this paper we tackle this challenge by synergizing reinforcement learning with principal-agent theory from economics. Taken separately, the former allows unrealistic freedom of intervention, while the latter struggles to scale in sequential settings. Combining them achieves the best of both worlds. We propose a framework where a principal guides an agent in a Markov Decision Process (MDP) using a series of contracts, which specify payments by the principal based on observable outcomes of the agent's actions. We present and analyze a meta-algorithm that iteratively optimizes the policies of the principal and agent, showing its equivalence to a contraction operator on the principal's Q-function, and its convergence to subgame-perfect equilibrium. We then scale our algorithm with deep Q-learning and analyze its convergence in the presence of approximation error, both theoretically and through experiments with randomly generated binary game-trees. Extending our framework to multiple agents, we apply our methodology to the combinatorial Coin Game. Addressing this multi-agent sequential social dilemma is a promising first step toward scaling our approach to more complex, real-world instances.

Figures (4)

• Figure 1: Example of a principal-agent MDP with three states $S = \{s_0,s_L,s_R\}.$ In each state, the agent can take one of two actions: noisy-left $a_L$, which is costly and leads to outcomes $L$ and $R$ with probabilities $0.9$ and $0.1$, and noisy-right $a_R$, which is free and has the likelihood of $L$ and $R$ reversed. The principal's rewards in any state $s \in S$ for outcomes $L,R$ are $r^p(s,L)= \frac{14}{9},r^p(s,R)=0$, while those of the agent for the actions are $r(s,a_L)=-\frac{4}{5},r(s,a_R)=0$. For analysis, see \ref{['app:proof_example_revisited']}.
• Figure 2: Learning curves in the Coin Game. See \ref{['sec:multi_agent_experiments']} for plot explanations. Shaded regions represent standard errors in the top plots and min-max ranges in the bottom plots.
• Figure 3: Results in Tree MDPs. Solid lines are learning curves of DQNs trained with Algorithm \ref{['alg:single_agent']}; the utilities are computed by coupling the learned principal's policy with a best-responding oracle agent throughout the training. Dashed lines represent optimal utilities in SPE obtained with dynamic programming. Different colors represent three distinct instances of the tree environment. For each, we use five trials of the algorithm (shaded regions represent standard errors).
• Figure 4: Learning curves in the Coin Game with a $3 \times 3$ grid and each episode lasting for $20$ time steps. The structure is the same as in \ref{['fig:multi_agent']}, with the addition of the top middle plot. The definition of the plots is as follows: a) total return of the two agents (without payments); b) same, but our algorithm and constant baseline additionally pay $10\%$ of social welfare to the agents; c) a ratio of the total payment by the principal to what the agents would effectively be paid if directly maximizing social welfare; d) the proportion of the principal's recommendations followed by the validation agents; r) a ratio between the utilities of an agent's policy at a given iteration and the recommended policy, with the opponent using the recommended policy; f) same ratio, but with the opponent's policy at a given iteration. Each experiment is repeated 5 times, and each measurement is averaged over 80 episodes.

Theorems & Definitions (29)

• Definition 3.2
• Theorem 3.3
• Theorem 3.4
• Proposition 4.1
• Definition B.1
• Definition B.2
• Lemma B.3
• proof
• Definition B.4
• Definition B.5