Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Off-Policy Correction For Multi-Agent Reinforcement Learning

Michał Zawalski, Błażej Osiński, Henryk Michalewski, Piotr Miłoś

TL;DR

MA-Trace extends V-Trace to multi-agent reinforcement learning under centralized training and decentralized execution, enabling scalable multi-node training through off-policy corrections via importance sampling. The authors establish a fixed-point contraction for the MA-Trace operator, guaranteeing convergence to a policy that interpolates between the behavior and target policies. Empirically, MA-Trace demonstrates competitive performance on the StarCraft Multi-Agent Challenge and scales effectively with the number of actor workers, with ablations illustrating the critical role of importance sampling and critic input choice. The work provides both theoretical grounding and practical insights for scalable MARL, highlighting trade-offs between centralized and decentralized training and the benefits of shared actor networks. Overall, MA-Trace offers a principled, scalable approach for distributed MARL with strong empirical validation.

Abstract

Multi-agent reinforcement learning (MARL) provides a framework for problems involving multiple interacting agents. Despite apparent similarity to the single-agent case, multi-agent problems are often harder to train and analyze theoretically. In this work, we propose MA-Trace, a new on-policy actor-critic algorithm, which extends V-Trace to the MARL setting. The key advantage of our algorithm is its high scalability in a multi-worker setting. To this end, MA-Trace utilizes importance sampling as an off-policy correction method, which allows distributing the computations with no impact on the quality of training. Furthermore, our algorithm is theoretically grounded - we prove a fixed-point theorem that guarantees convergence. We evaluate the algorithm extensively on the StarCraft Multi-Agent Challenge, a standard benchmark for multi-agent algorithms. MA-Trace achieves high performance on all its tasks and exceeds state-of-the-art results on some of them.

Off-Policy Correction For Multi-Agent Reinforcement Learning

TL;DR

MA-Trace extends V-Trace to multi-agent reinforcement learning under centralized training and decentralized execution, enabling scalable multi-node training through off-policy corrections via importance sampling. The authors establish a fixed-point contraction for the MA-Trace operator, guaranteeing convergence to a policy that interpolates between the behavior and target policies. Empirically, MA-Trace demonstrates competitive performance on the StarCraft Multi-Agent Challenge and scales effectively with the number of actor workers, with ablations illustrating the critical role of importance sampling and critic input choice. The work provides both theoretical grounding and practical insights for scalable MARL, highlighting trade-offs between centralized and decentralized training and the benefits of shared actor networks. Overall, MA-Trace offers a principled, scalable approach for distributed MARL with strong empirical validation.

Abstract

Multi-agent reinforcement learning (MARL) provides a framework for problems involving multiple interacting agents. Despite apparent similarity to the single-agent case, multi-agent problems are often harder to train and analyze theoretically. In this work, we propose MA-Trace, a new on-policy actor-critic algorithm, which extends V-Trace to the MARL setting. The key advantage of our algorithm is its high scalability in a multi-worker setting. To this end, MA-Trace utilizes importance sampling as an off-policy correction method, which allows distributing the computations with no impact on the quality of training. Furthermore, our algorithm is theoretically grounded - we prove a fixed-point theorem that guarantees convergence. We evaluate the algorithm extensively on the StarCraft Multi-Agent Challenge, a standard benchmark for multi-agent algorithms. MA-Trace achieves high performance on all its tasks and exceeds state-of-the-art results on some of them.
Paper Structure (36 sections, 3 theorems, 15 equations, 16 figures, 3 tables, 1 algorithm)

This paper contains 36 sections, 3 theorems, 15 equations, 16 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Background
  4. MA-Trace algorithm
  5. Overview of the algorithm
  6. Theoretical analysis of MA-Trace
  7. Experiments
  8. Environment
  9. Main result
  10. Training details
  11. Ablations
  12. Advantage of using importance sampling.
  13. Training scaling.
  14. Input for the critic network.
  15. Centralized vs decentralized.
  16. ...and 21 more sections

Key Result

Theorem 1

Let $c, \rho$ be such that for any $s \in \mathcal{S}, a\in \mathcal{A}$ where $s'$ is the state obtained from $s$ after issuing action $a$. Assume also that $\mathop{\mathrm{\mathbb{E}}}\nolimits_{\mu}\rho_{0}\geq\beta\in(0,1]$. Then the operator $\mathcal{R}$ is a $\mathcal{C}_{\infty}$ contraction with a unique fixed point $V^{\tilde{\pi}}$ which is a value function o The contraction constant i

Figures (16)

  • Figure 1: MA-Trace compared with state-of-the-arts algorithms on SMAC.
  • Figure 2: MA-Trace using 30 distributed workers with and without importance sampling (no IS).
  • Figure 3: Speed of MA-Trace training with respect to the number of distributed workers, with standard deviation shaded. The speed is measured as the average number of steps processed per second.
  • Figure 4: Comparison of using the full state MA-Trace (full) and aggregated agents' observations MA-Trace (obs) and both.
  • Figure 5: Performance of MA-Trace during centralized and decentralized training.
  • ...and 11 more figures

Theorems & Definitions (6)

  • Theorem 1
  • Remark 2
  • Corollary 3
  • Remark 4
  • Theorem 5
  • Remark 6