Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent Guided Policy Optimization

Yueheng Li, Guangming Xie, Zongqing Lu

Abstract

Due to practical constraints such as partial observability and limited communication, Centralized Training with Decentralized Execution (CTDE) has become the dominant paradigm in cooperative Multi-Agent Reinforcement Learning (MARL). However, existing CTDE methods often underutilize centralized training or lack theoretical guarantees. We propose Multi-Agent Guided Policy Optimization (MAGPO), a novel framework that better leverages centralized training by integrating centralized guidance with decentralized execution. MAGPO uses an autoregressive joint policy for scalable, coordinated exploration and explicitly aligns it with decentralized policies to ensure deployability under partial observability. We provide theoretical guarantees of monotonic policy improvement and empirically evaluate MAGPO on 43 tasks across 6 diverse environments. Results show that MAGPO consistently outperforms strong CTDE baselines and matches or surpasses fully centralized approaches, offering a principled and practical solution for decentralized multi-agent learning. Our code and experimental data can be found in https://github.com/liyheng/MAGPO.

Multi-Agent Guided Policy Optimization

Abstract

Due to practical constraints such as partial observability and limited communication, Centralized Training with Decentralized Execution (CTDE) has become the dominant paradigm in cooperative Multi-Agent Reinforcement Learning (MARL). However, existing CTDE methods often underutilize centralized training or lack theoretical guarantees. We propose Multi-Agent Guided Policy Optimization (MAGPO), a novel framework that better leverages centralized training by integrating centralized guidance with decentralized execution. MAGPO uses an autoregressive joint policy for scalable, coordinated exploration and explicitly aligns it with decentralized policies to ensure deployability under partial observability. We provide theoretical guarantees of monotonic policy improvement and empirically evaluate MAGPO on 43 tasks across 6 diverse environments. Results show that MAGPO consistently outperforms strong CTDE baselines and matches or surpasses fully centralized approaches, offering a principled and practical solution for decentralized multi-agent learning. Our code and experimental data can be found in https://github.com/liyheng/MAGPO.

Paper Structure

This paper contains 35 sections, 5 theorems, 30 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Formulation
  4. Related Works
  5. CTDE.
  6. CTDS.
  7. HARL.
  8. CTCE.
  9. Problems of Current CTDS
  10. Method
  11. Multi-Agent Guided Policy Optimization
  12. Practical Implementation
  13. Guider Update.
  14. Learner Update.
  15. Experiments
  16. ...and 20 more sections

Key Result

Theorem 4.1

Let $(\bm{\pi}_k)_{k=0}^\infty$ be the sequence of joint learner policies obtained by iteratively applying the four steps of MAGPO. Then, where $V_\rho$ is the expected return under initial state distribution $\rho$.

Figures (7)

  • Figure 1: Illustrative example showing three different MARL settings.
  • Figure 2: The sample efficiency curves aggregated per environment suite, where dashed lines represent the CTCE methods. For each environment, results are aggregated over all tasks and the min–max normalized inter-quartile mean with 95% stratified bootstrap confidence intervals are shown.
  • Figure 3: The overall aggregated probability of improvement for MAGPO compared to other baselines for that specific environment. A score of more than 0.5 where confidence intervals are also greater than 0.5 indicates statistically significant improvement over a baseline for a given environment agarwal2021deep.
  • Figure 4: MAGPO performance varies with the choice of guider and the regularization ratio $\delta$.
  • Figure 5: The effect of RL auxiliary loss.
  • ...and 2 more figures

Theorems & Definitions (9)

  • Theorem 4.1: Monotonic Improvement of MAGPO
  • proof
  • Lemma 1: Multi-Agent Advantage Decomposition kuba2022trustregionpolicyoptimisation
  • Corollary 4.2: Sequential Update of MAGPO
  • proof
  • Theorem A.1: Monotonic Improvement of MAGPO
  • proof
  • Corollary A.2: Sequential Update of MAGPO
  • proof