Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization

Zongkai Liu, Qian Lin, Chao Yu, Xiawei Wu, Yile Liang, Donghui Li, Xuetao Ding

TL;DR

This work tackles offline MARL challenges of distribution shift and OOD joint actions by introducing InSPO, an in-sample, sequential policy optimization method built on a Maximum-Entropy Behavior-Regularized Markov Game framework. InSPO updates agents one-by-one using a reverse KL regularizer, derives a closed-form in-sample policy update, and employs local Q-functions with importance resampling to maintain data-consistency. The authors prove monotonic policy improvement and convergence to a quantal response equilibrium (QRE), and demonstrate strong empirical performance on XOR, M-NE, Bridge, and StarCraft II benchmarks, often outperforming state-of-the-art offline MARL baselines. The approach enhances coordination under offline data constraints and provides a principled, theoretically grounded path toward robust multi-agent learning without generating out-of-distribution joint actions.

Abstract

Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large joint state-action space and coupled behaviors of multiple agents, which bring extra complexity to offline policy optimization. In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic, leading to uncoordinated behaviors and out-of-distribution (OOD) joint actions. To address these issues, we propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO). InSPO sequentially updates each agent's policy in an in-sample manner, which not only avoids selecting OOD joint actions but also carefully considers teammates' updated policies to enhance coordination. Additionally, by thoroughly exploring low-probability actions in the behavior policy, InSPO can well address the issue of premature convergence to sub-optimal solutions. Theoretically, we prove InSPO guarantees monotonic policy improvement and converges to quantal response equilibrium (QRE). Experimental results demonstrate the effectiveness of our method compared to current state-of-the-art offline MARL methods.

Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization

TL;DR

This work tackles offline MARL challenges of distribution shift and OOD joint actions by introducing InSPO, an in-sample, sequential policy optimization method built on a Maximum-Entropy Behavior-Regularized Markov Game framework. InSPO updates agents one-by-one using a reverse KL regularizer, derives a closed-form in-sample policy update, and employs local Q-functions with importance resampling to maintain data-consistency. The authors prove monotonic policy improvement and convergence to a quantal response equilibrium (QRE), and demonstrate strong empirical performance on XOR, M-NE, Bridge, and StarCraft II benchmarks, often outperforming state-of-the-art offline MARL baselines. The approach enhances coordination under offline data constraints and provides a principled, theoretically grounded path toward robust multi-agent learning without generating out-of-distribution joint actions.

Abstract

Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large joint state-action space and coupled behaviors of multiple agents, which bring extra complexity to offline policy optimization. In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic, leading to uncoordinated behaviors and out-of-distribution (OOD) joint actions. To address these issues, we propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO). InSPO sequentially updates each agent's policy in an in-sample manner, which not only avoids selecting OOD joint actions but also carefully considers teammates' updated policies to enhance coordination. Additionally, by thoroughly exploring low-probability actions in the behavior policy, InSPO can well address the issue of premature convergence to sub-optimal solutions. Theoretically, we prove InSPO guarantees monotonic policy improvement and converges to quantal response equilibrium (QRE). Experimental results demonstrate the effectiveness of our method compared to current state-of-the-art offline MARL methods.

Paper Structure

This paper contains 40 sections, 9 theorems, 52 equations, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. MARL.
  4. Offline MARL.
  5. Background
  6. Cooperative Markov Game
  7. IGM Principle and Value Decomposition
  8. Behavior-Regularized Markov Game in Offline MARL
  9. The Proposed Method
  10. OOD Joint Action in Offline MARL
  11. In-Sample Sequential Policy Optimization
  12. Maximum-Entropy Behavior-Regularized Markov Game
  13. The Practical Implementation of InSPO
  14. Policy Evaluation.
  15. Policy Improvement.
  16. ...and 25 more sections

Key Result

Lemma 2

Given a policy $\boldsymbol{\pi}$, consider the modified policy evaluation operator $\mathcal{T}_{\boldsymbol{\pi}}$ in Eq.(eq: policy evaluation operator) and a initial Q-function $\boldsymbol{Q}_0: \mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and define $\boldsymbol{Q}_{k+1}=\mathcal{T}_{\b

Figures (5)

  • Figure 1: XOR game. (a) is the reward matrix of joint actions. (b) is the distribution of dataset.
  • Figure 2: M-NE game. (a) is the reward matrix of joint actions. (b) is the distribution of dataset.
  • Figure 3: Final joint policy on XOR game for dataset (b).
  • Figure 4: Bridge at the beginning.
  • Figure 5: Ablation on entropy and sequential update scheme. (a) is InSPO without entropy on M-NE game for the imbalanced dataset. (b) is simultaneous-update version of InSPO on XOR game for dataset (b).

Theorems & Definitions (15)

  • Definition 1
  • Lemma 2
  • Proposition 3
  • Proposition 4
  • Theorem 5
  • Lemma 6
  • proof
  • Proposition 7
  • proof
  • Proposition 8
  • ...and 5 more