Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Problem of Social Cost in Multi-Agent General Reinforcement Learning: Survey and Synthesis

Kee Siong Ng, Samuel Yang-Zhao, Timothy Cadogan-Cowper

TL;DR

This paper reframes safety challenges in multi-agent general reinforcement learning as social-cost problems arising from externalities among learning agents. It develops a market-based, mechanism-design framework—centered on VCG and Exponential VCG—that internalises social harms by taxing or pricing the externalities each action imposes on others, within a history-based GRL setting. The authors integrate Bayesian reinforcement learning, online mixture models, Monte Carlo planning, and dynamic hedging to enable learning of valuation and social-cost functions, with theoretical properties like incentive compatibility and approximate equilibrium under various conditions. They illustrate the approach with applications such as Paperclips and cap-and-trade for pollution, highlighting how mechanism design can steer complex, long-horizon multi-agent systems toward socially beneficial outcomes while addressing practical learning and privacy considerations. Overall, the work provides a principled bridge between AI safety, economics, and MARL, offering a scalable template for regulating powerful agents through market-like social costs in dynamic environments.

Abstract

The AI safety literature is full of examples of powerful AI agents that, in blindly pursuing a specific and usually narrow objective, ends up with unacceptable and even catastrophic collateral damage to others. In this paper, we consider the problem of social harms that can result from actions taken by learning and utility-maximising agents in a multi-agent environment. The problem of measuring social harms or impacts in such multi-agent settings, especially when the agents are artificial generally intelligent (AGI) agents, was listed as an open problem in Everitt et al, 2018. We attempt a partial answer to that open problem in the form of market-based mechanisms to quantify and control the cost of such social harms. The proposed setup captures many well-studied special cases and is more general than existing formulations of multi-agent reinforcement learning with mechanism design in two ways: (i) the underlying environment is a history-based general reinforcement learning environment like in AIXI; (ii) the reinforcement-learning agents participating in the environment can have different learning strategies and planning horizons. To demonstrate the practicality of the proposed setup, we survey some key classes of learning algorithms and present a few applications, including a discussion of the Paperclips problem and pollution control with a cap-and-trade system.

The Problem of Social Cost in Multi-Agent General Reinforcement Learning: Survey and Synthesis

TL;DR

This paper reframes safety challenges in multi-agent general reinforcement learning as social-cost problems arising from externalities among learning agents. It develops a market-based, mechanism-design framework—centered on VCG and Exponential VCG—that internalises social harms by taxing or pricing the externalities each action imposes on others, within a history-based GRL setting. The authors integrate Bayesian reinforcement learning, online mixture models, Monte Carlo planning, and dynamic hedging to enable learning of valuation and social-cost functions, with theoretical properties like incentive compatibility and approximate equilibrium under various conditions. They illustrate the approach with applications such as Paperclips and cap-and-trade for pollution, highlighting how mechanism design can steer complex, long-horizon multi-agent systems toward socially beneficial outcomes while addressing practical learning and privacy considerations. Overall, the work provides a principled bridge between AI safety, economics, and MARL, offering a scalable template for regulating powerful agents through market-like social costs in dynamic environments.

Abstract

The AI safety literature is full of examples of powerful AI agents that, in blindly pursuing a specific and usually narrow objective, ends up with unacceptable and even catastrophic collateral damage to others. In this paper, we consider the problem of social harms that can result from actions taken by learning and utility-maximising agents in a multi-agent environment. The problem of measuring social harms or impacts in such multi-agent settings, especially when the agents are artificial generally intelligent (AGI) agents, was listed as an open problem in Everitt et al, 2018. We attempt a partial answer to that open problem in the form of market-based mechanisms to quantify and control the cost of such social harms. The proposed setup captures many well-studied special cases and is more general than existing formulations of multi-agent reinforcement learning with mechanism design in two ways: (i) the underlying environment is a history-based general reinforcement learning environment like in AIXI; (ii) the reinforcement-learning agents participating in the environment can have different learning strategies and planning horizons. To demonstrate the practicality of the proposed setup, we survey some key classes of learning algorithms and present a few applications, including a discussion of the Paperclips problem and pollution control with a cap-and-trade system.

Paper Structure

This paper contains 61 sections, 13 theorems, 98 equations, 8 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. General Reinforcement Learning
  3. Single Agent Setting
  4. Multi-Agent Setting
  5. Mechanism Design
  6. Tragedy of the Commons
  7. The VCG Mechanism
  8. The Exponential VCG Mechanism
  9. The Social Cost of Actions
  10. General Case
  11. Variations of Total Social Welfare
  12. Variations of Agent Valuation Functions
  13. Variations of Payment Functions
  14. Partial Observability
  15. Private Information
  16. ...and 46 more sections

Key Result

Theorem 1

Hutter:04uaibook Let $\mu$ be the true environment and $\xi$ be the Bayesian mixture model over a model class $\mathcal{M}$. For all $n \in \mathbb{N}$ and for all $a_{1:n}$, where $D_n(\mu || \rho)$ is the KL divergence of $\mu(\cdot | a_{1:n})$ and $\rho(\cdot | a_{1:n})$ defined by

Figures (8)

  • Figure 1: A horizon-2 game tree with small action and percept spaces
  • Figure 2: Plots of $s$ and $s^{-1}$ for refineries $R_{1}$ and $R_{2}$.
  • Figure 3: Cubic tons of greenhouse gas emitted for different production of fuel
  • Figure 4: RL with reward function $r^{(1)}$ (incentivise individual profit)
  • Figure 5: RL with reward function $r^{(2)}$ (incentivise joint profit)
  • ...and 3 more figures

Theorems & Definitions (45)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Theorem 1
  • Definition 5
  • Definition 6
  • Definition 7
  • Definition 8
  • Definition 9
  • ...and 35 more