Dynamics of Multi-Agent Actor-Critic Learning in Stochastic Games: from Multistability and Chaos to Stable Cooperation

TL;DR

This paper develops a dynamical systems framework for entropy-regularized actor-critic learning in stochastic games, deriving continuous-time ODEs via two-timescale separation to study policy evolution. It reveals chaotic dynamics in the two-state Matching Pennies without regularization, while entropy regularization induces dissipativity that guides the system toward stable, fair cooperation; in the two-state Prisoner’s Dilemma, it uncovers multistability corresponding to ALLC, ALLD, and GRIM, with regularization enlarging the cooperative basin. A key contribution is the connection between direct reciprocity mechanisms in evolutionary game theory and cooperation emergence in MARL, offering principled guidance for designing resilient multi-agent systems. The results bridge MARL and EGT theory and highlight practical tuning of entropy regularization to promote robust cooperation in dynamic, multi-agent environments.

Abstract

Achieving robust coordination and cooperation is a central challenge in multi-agent reinforcement learning (MARL). Uncovering the mechanisms underlying such emergent behaviors calls for a dynamical understanding of learn processes. In this work, we investigate the dynamics of actor-critic agents in stochastic games, focusing on the impact of entropy regularization. By leveraging time-scale separation, we derive the system's evolution equations, which are then formally analyzed using dynamical systems theory. We find that in the constant-sum game of Matching Pennies, the system exhibits chaotic behavior. Entropy regularization mitigates this chaos and drives the dynamics toward convergence to fair cooperation. In contrast, in the general-sum game of the Prisoner's Dilemma, the system displays multistability. Interestingly, the three stable equilibria of the system correspond to the well-known ALLC (Always Cooperate), ALLD (Always Defect), and GRIM (Grim Trigger) strategies from evolutionary game theory (EGT). Entropy regularization strengthens system resilience by enlarging the basin of attraction of the cooperative equilibrium. Our findings reveal a close link between the mechanism of direct reciprocity in EGT and how cooperation emerges in MARL, offering insights for designing more robust and collaborative multi-agent systems.

Figures (6)

• Figure 1: Policy evolution in the two-state Matching Pennies game without entropy regularization. For each parameter combination, the subfigure displays the trajectories of the two players' policies in state $s_1$ and $s_2$, respectively.
• Figure 2: Maximum Lyapunov exponent (MLE) of the two-state Matching Pennies game without entropy regularization. The scattered points represent the MLE values for different values of $\gamma$, quantifying the sensitivity of agents' trajectories to small perturbations in initial conditions.
• Figure 3: Policy evolution in the two-state Matching Pennies game with entropy regularization. Each subfigure shows ten simulation trajectories of the two players' policies in state $s_1$ and $s_2$, respectively, with trajectories initialized from random starting points.
• Figure 4: Vector fields and numerical simulation trajectories for the two-state Prisoner's Dilemma game without entropy regularization. The x-axis is the cooperation probability of both agents in state $s_1$, and the y-axis is the cooperation probability in state $s_2$.
• Figure 5: Finite-state automaton representations of three classical strategies in the iterated Prisoner's Dilemma binmore1992evolutionary. Each node represents an agent's internal state (distinct from environmental states) and prescribes a specific action. The letters C and D denote cooperation (action $a_1$) and defection (action $a_2$), respectively. The arrows indicate transitions between internal states based on the opponent's previous action. For the GRIM strategy, the initial internal state is C.

Theorems & Definitions (2)

• Theorem 1
• Theorem 2