Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Experience-replay Innovative Dynamics

Tuo Zhang, Leonardo Stella, Julian Barreiro-Gomez

TL;DR

This work tackles instability and nonstationarity in multi-agent reinforcement learning by moving beyond traditional replicator dynamics to innovative dynamics that can converge to Nash equilibria in broader game classes. It introduces Experience-replay Innovative Dynamics (ERID), a stateless reinforcement learning framework that uses an experience replay buffer and revision-protocol-based updates to emulate Brown-von Neumann-Nash (BNN), Smith, and Smith-replicator dynamics via adaptable protocol factors. The authors prove that ERID trajectories converge to the target dynamic using limits such as $\alpha K \to 0$ and $K \to \infty$, and validate these results with experiments on Matching Pennies and biased Rock-Paper-Scissors, demonstrating robust adaptation to nonstationary environments. Overall, ERID extends theoretical convergence guarantees for MARL beyond replicator dynamics and offers a practical mechanism to maintain equilibrium convergence under changing conditions.

Abstract

Despite its groundbreaking success, multi-agent reinforcement learning (MARL) still suffers from instability and nonstationarity. Replicator dynamics, the most well-known model from evolutionary game theory (EGT), provide a theoretical framework for the convergence of the trajectories to Nash equilibria and, as a result, have been used to ensure formal guarantees for MARL algorithms in stable game settings. However, they exhibit the opposite behavior in other settings, which poses the problem of finding alternatives to ensure convergence. In contrast, innovative dynamics, such as the Brown-von Neumann-Nash (BNN) or Smith, result in periodic trajectories with the potential to approximate Nash equilibria. Yet, no MARL algorithms based on these dynamics have been proposed. In response to this challenge, we develop a novel experience replay-based MARL algorithm that incorporates revision protocols as tunable hyperparameters. We demonstrate, by appropriately adjusting the revision protocols, that the behavior of our algorithm mirrors the trajectories resulting from these dynamics. Importantly, our contribution provides a framework capable of extending the theoretical guarantees of MARL algorithms beyond replicator dynamics. Finally, we corroborate our theoretical findings with empirical results.

Experience-replay Innovative Dynamics

TL;DR

This work tackles instability and nonstationarity in multi-agent reinforcement learning by moving beyond traditional replicator dynamics to innovative dynamics that can converge to Nash equilibria in broader game classes. It introduces Experience-replay Innovative Dynamics (ERID), a stateless reinforcement learning framework that uses an experience replay buffer and revision-protocol-based updates to emulate Brown-von Neumann-Nash (BNN), Smith, and Smith-replicator dynamics via adaptable protocol factors. The authors prove that ERID trajectories converge to the target dynamic using limits such as and , and validate these results with experiments on Matching Pennies and biased Rock-Paper-Scissors, demonstrating robust adaptation to nonstationary environments. Overall, ERID extends theoretical convergence guarantees for MARL beyond replicator dynamics and offers a practical mechanism to maintain equilibrium convergence under changing conditions.

Abstract

Despite its groundbreaking success, multi-agent reinforcement learning (MARL) still suffers from instability and nonstationarity. Replicator dynamics, the most well-known model from evolutionary game theory (EGT), provide a theoretical framework for the convergence of the trajectories to Nash equilibria and, as a result, have been used to ensure formal guarantees for MARL algorithms in stable game settings. However, they exhibit the opposite behavior in other settings, which poses the problem of finding alternatives to ensure convergence. In contrast, innovative dynamics, such as the Brown-von Neumann-Nash (BNN) or Smith, result in periodic trajectories with the potential to approximate Nash equilibria. Yet, no MARL algorithms based on these dynamics have been proposed. In response to this challenge, we develop a novel experience replay-based MARL algorithm that incorporates revision protocols as tunable hyperparameters. We demonstrate, by appropriately adjusting the revision protocols, that the behavior of our algorithm mirrors the trajectories resulting from these dynamics. Importantly, our contribution provides a framework capable of extending the theoretical guarantees of MARL algorithms beyond replicator dynamics. Finally, we corroborate our theoretical findings with empirical results.
Paper Structure (15 sections, 4 theorems, 18 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 15 sections, 4 theorems, 18 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. Preliminaries
  4. Game Theory
  5. Dynamics and Learning
  6. Experience-replay Innovative Dynamics
  7. Experience Replay
  8. Experience Replayed Cross Learning
  9. ERID with BNN Dynamics
  10. ERID with Smith Dynamics
  11. ERID with Smith-replicator-based Pairwise Dynamics
  12. Evaluation
  13. Comparison Dynamics with ERID
  14. Comparison Cross Learning with ERID
  15. Conclusion

Key Result

Lemma 1

Let us consider a cross learning model in a 2-player normal form game. If $\alpha \rightarrow 0$, then the trajectory of cross learning converges to the trajectory of the two-player normal form game resulting from replicator dynamics.

Figures (4)

  • Figure 1: Policy NashConv of BNN, Replicator and Hedge algorithms in nonstationary RPS, with the game phases every 3000 iterations separated by vertical red lines.
  • Figure 2: Innovative dynamics (left) vs policy trajectories (right) of the ERID algorithm using BNN dynamics (top) and Smith dynamics (bottom), respectively.
  • Figure 3: Innovative dynamics (left) vs policy trajectories (right) of the ERID algorithm using BNN dynamics (top) and Smith dynamics (bottom), respectively.
  • Figure 4: Policy NashConv and relative NashConv of ERID with BNN, ERID with Smith, Cross learning in nonstationary RPS.

Theorems & Definitions (4)

  • Lemma 1
  • Theorem 1
  • Theorem 2
  • Theorem 3