Maximum Entropy Heterogeneous-Agent Reinforcement Learning

TL;DR

This work tackles the instability and suboptimal convergence in cooperative MARL by proposing a Maximum Entropy MARL framework that casts learning as probabilistic inference. It derives a MaxEnt objective, connects stochastic policies to the quantal response equilibrium, and introduces HASAC with a MEHAML template that guarantees monotonic improvement and convergence to QRE. The approach combines centralized soft value learning with sequential, stochastic policy updates across heterogeneous agents, yielding improved sample efficiency, robustness, and exploration across diverse continuous and discrete tasks. Empirical results on six benchmarks (e.g., SMAC, Bi-DexHands, MAMuJoCo, GRF) show HASAC consistently outperforms strong baselines, often avoiding premature convergence to suboptimal equilibria. Overall, MEHARL offers a principled, scalable path to stable, exploratory multi-agent collaboration with theoretical guarantees and broad applicability.

Abstract

Multi-agent reinforcement learning (MARL) has been shown effective for cooperative games in recent years. However, existing state-of-the-art methods face challenges related to sample complexity, training instability, and the risk of converging to a suboptimal Nash Equilibrium. In this paper, we propose a unified framework for learning stochastic policies to resolve these issues. We embed cooperative MARL problems into probabilistic graphical models, from which we derive the maximum entropy (MaxEnt) objective for MARL. Based on the MaxEnt framework, we propose Heterogeneous-Agent Soft Actor-Critic (HASAC) algorithm. Theoretically, we prove the monotonic improvement and convergence to quantal response equilibrium (QRE) properties of HASAC. Furthermore, we generalize a unified template for MaxEnt algorithmic design named Maximum Entropy Heterogeneous-Agent Mirror Learning (MEHAML), which provides any induced method with the same guarantees as HASAC. We evaluate HASAC on six benchmarks: Bi-DexHands, Multi-Agent MuJoCo, StarCraft Multi-Agent Challenge, Google Research Football, Multi-Agent Particle Environment, and Light Aircraft Game. Results show that HASAC consistently outperforms strong baselines, exhibiting better sample efficiency, robustness, and sufficient exploration. See our page at https://sites.google.com/view/meharl.

Figures (11)

• Figure 1: A single-state 2-agent cooperative matrix game. (a) is the reward matrix of joint actions. (b) represents the initial joint policy $\boldsymbol{\pi}$ formed by both agents taking the individual policy $\pi = \{0.6, 0.2, 0.2\}$. (c) represents the final joint policy $\boldsymbol{\pi}$ that MAPPO and HAPPO converge to, deterministically choosing action $(A, A)$.
• Figure 2: The probabilistic graphical model for cooperative MARL.
• Figure 3: Performance comparisons on selected tasks of multiple benchmarks.
• Figure 4: Performance comparison between HASAC with different hyperparameters on Ant-v2 4x2 task and matrix game. (a) The comparison indicates that stochastic policy can lead to better equilibrium and stabilize training. (b) & (c) HASAC converges to different QRE with different temperature parameter $\alpha$.
• Figure 5: Different temperature $\alpha$ leads to convergence of different QRE.

Theorems & Definitions (35)

• Theorem 1: Representation of QRE
• Lemma 4.1: Joint Soft Policy Evaluation
• Definition 1
• Proposition 1: Joint Soft Policy Decomposition
• Lemma 4.2: Heterogeneous-Agent Soft Policy Improvement
• Theorem 2: Heterogeneous-Agent Soft Policy Iteration
• Definition 2
• Theorem 3: The Core Theorem of MEHAML
• Definition 3: IGO
• Definition 4