Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Backpropagation Through Agents

Zhiyuan Li, Wenshuai Zhao, Lijun Wu, Joni Pajarinen

TL;DR

Backpropagation Through Agents (BPTA) introduces bidirectional gradients in multi-agent reinforcement learning by propagating feedback through action chains, addressing the limitations of forward-only auto-regressive policies. By deriving a conditional multi-agent stochastic policy gradient theorem and incorporating a peer-learning term via the reparameterization trick, BPTA enables agents to adapt based on downstream reactions. BPPO, the PPO-based instantiation of BPTA, demonstrates superior performance across matrix games, SMACv2, MA-MuJoCo, and Google Football benchmarks, highlighting improved coordination and robustness as team size grows. This framework offers a scalable path to coherent joint policies in complex, partially observable multi-agent environments.

Abstract

A fundamental challenge in multi-agent reinforcement learning (MARL) is to learn the joint policy in an extremely large search space, which grows exponentially with the number of agents. Moreover, fully decentralized policy factorization significantly restricts the search space, which may lead to sub-optimal policies. In contrast, the auto-regressive joint policy can represent a much richer class of joint policies by factorizing the joint policy into the product of a series of conditional individual policies. While such factorization introduces the action dependency among agents explicitly in sequential execution, it does not take full advantage of the dependency during learning. In particular, the subsequent agents do not give the preceding agents feedback about their decisions. In this paper, we propose a new framework Back-Propagation Through Agents (BPTA) that directly accounts for both agents' own policy updates and the learning of their dependent counterparts. This is achieved by propagating the feedback through action chains. With the proposed framework, our Bidirectional Proximal Policy Optimisation (BPPO) outperforms the state-of-the-art methods. Extensive experiments on matrix games, StarCraftII v2, Multi-agent MuJoCo, and Google Research Football demonstrate the effectiveness of the proposed method.

Backpropagation Through Agents

TL;DR

Backpropagation Through Agents (BPTA) introduces bidirectional gradients in multi-agent reinforcement learning by propagating feedback through action chains, addressing the limitations of forward-only auto-regressive policies. By deriving a conditional multi-agent stochastic policy gradient theorem and incorporating a peer-learning term via the reparameterization trick, BPTA enables agents to adapt based on downstream reactions. BPPO, the PPO-based instantiation of BPTA, demonstrates superior performance across matrix games, SMACv2, MA-MuJoCo, and Google Football benchmarks, highlighting improved coordination and robustness as team size grows. This framework offers a scalable path to coherent joint policies in complex, partially observable multi-agent environments.

Abstract

A fundamental challenge in multi-agent reinforcement learning (MARL) is to learn the joint policy in an extremely large search space, which grows exponentially with the number of agents. Moreover, fully decentralized policy factorization significantly restricts the search space, which may lead to sub-optimal policies. In contrast, the auto-regressive joint policy can represent a much richer class of joint policies by factorizing the joint policy into the product of a series of conditional individual policies. While such factorization introduces the action dependency among agents explicitly in sequential execution, it does not take full advantage of the dependency during learning. In particular, the subsequent agents do not give the preceding agents feedback about their decisions. In this paper, we propose a new framework Back-Propagation Through Agents (BPTA) that directly accounts for both agents' own policy updates and the learning of their dependent counterparts. This is achieved by propagating the feedback through action chains. With the proposed framework, our Bidirectional Proximal Policy Optimisation (BPPO) outperforms the state-of-the-art methods. Extensive experiments on matrix games, StarCraftII v2, Multi-agent MuJoCo, and Google Research Football demonstrate the effectiveness of the proposed method.
Paper Structure (27 sections, 3 theorems, 14 equations, 13 figures, 8 tables, 1 algorithm)

This paper contains 27 sections, 3 theorems, 14 equations, 13 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Problem Formulation
  5. Independent Multi-Agent Stochastic Policy Gradient
  6. Method
  7. Back-Propagation Through Agents
  8. Experiments
  9. Matrix Games
  10. SMACv2
  11. MA-MuJoCo
  12. GRF
  13. Conclusion
  14. Derivation of Multi-Agent Conditional Policy Gradient Theorem
  15. Experimental Details
  16. ...and 12 more sections

Key Result

Theorem 1

For any episodic cooperative stochastic game with n agents, the gradient of the expected total reward for agent $i$, who has a backward dependency on some other peer agents $\mathcal{B}_i$ using parameters $\theta_{\mathcal{B}_i}$, with respect to current policy parameters $\theta_i$ is: where $\mathcal{F}_{\mathcal{B}_i}$ indicates the set of agents on which $\mathcal{B}_i$ have forward dependen

Figures (13)

  • Figure 1: Learning framework of BPTA-augmented auto-regressive policy gradient algorithm. BPTA internally completes the causality loop by two processes: (1) the forward process, which generates the causal flow by action; and (2) the backward process, which inverts the causal flow by propagating the feedback.
  • Figure 2: Comparison of training results on SMACv2.
  • Figure 3: Performance comparison on multiple Multi-Agent MuJoCo tasks.
  • Figure 4: Performance comparison on the Climbing game and the Penalty game.
  • Figure 5: Averaged train win rate on the Google Research Football scenarios.
  • ...and 8 more figures

Theorems & Definitions (5)

  • Theorem 1: Conditional Multi-Agent Stochastic Policy Gradient Theorem
  • proof
  • Corollary 1.1: Clipping Objective of BPPO
  • Theorem 1: Conditional Multi-Agent Stochastic Policy Gradient Theorem
  • proof