Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-agent Off-policy Actor-Critic Reinforcement Learning for Partially Observable Environments

Ainur Zhaikhan, Ali H. Sayed

TL;DR

The paper tackles partial observability in decentralized multi-agent reinforcement learning by estimating a global state via social learning and integrating it into a multi-agent off-policy actor-critic framework.It extends MAOPAC to dec-POMDPs in a model-free fashion, proving that the state-estimation error can be kept $\varepsilon$-small under sufficient diffusion iterations and providing Boltzmann-policy–specific guarantees.The work combines diffusion-based belief updates with off-policy corrections (importance sampling and ETD) and offers rigorous convergence analyses for the proposed framework.Empirical results on grid-based Dec-POMDP tasks show the approach approaching the performance of fully observed MAOPAC and outperforming zeroth-order policy optimization baselines, while achieving better critic agreement across agents.Overall, the method offers a practical, model-free path to decentralized coordination under partial observability with theoretical performance bounds and competitive empirical results.

Abstract

This study proposes the use of a social learning method to estimate a global state within a multi-agent off-policy actor-critic algorithm for reinforcement learning (RL) operating in a partially observable environment. We assume that the network of agents operates in a fully-decentralized manner, possessing the capability to exchange variables with their immediate neighbors. The proposed design methodology is supported by an analysis demonstrating that the difference between final outcomes, obtained when the global state is fully observed versus estimated through the social learning method, is $\varepsilon$-bounded when an appropriate number of iterations of social learning updates are implemented. Unlike many existing dec-POMDP-based RL approaches, the proposed algorithm is suitable for model-free multi-agent reinforcement learning as it does not require knowledge of a transition model. Furthermore, experimental results illustrate the efficacy of the algorithm and demonstrate its superiority over the current state-of-the-art methods.

Multi-agent Off-policy Actor-Critic Reinforcement Learning for Partially Observable Environments

TL;DR

The paper tackles partial observability in decentralized multi-agent reinforcement learning by estimating a global state via social learning and integrating it into a multi-agent off-policy actor-critic framework.It extends MAOPAC to dec-POMDPs in a model-free fashion, proving that the state-estimation error can be kept $\varepsilon$-small under sufficient diffusion iterations and providing Boltzmann-policy–specific guarantees.The work combines diffusion-based belief updates with off-policy corrections (importance sampling and ETD) and offers rigorous convergence analyses for the proposed framework.Empirical results on grid-based Dec-POMDP tasks show the approach approaching the performance of fully observed MAOPAC and outperforming zeroth-order policy optimization baselines, while achieving better critic agreement across agents.Overall, the method offers a practical, model-free path to decentralized coordination under partial observability with theoretical performance bounds and competitive empirical results.

Abstract

This study proposes the use of a social learning method to estimate a global state within a multi-agent off-policy actor-critic algorithm for reinforcement learning (RL) operating in a partially observable environment. We assume that the network of agents operates in a fully-decentralized manner, possessing the capability to exchange variables with their immediate neighbors. The proposed design methodology is supported by an analysis demonstrating that the difference between final outcomes, obtained when the global state is fully observed versus estimated through the social learning method, is -bounded when an appropriate number of iterations of social learning updates are implemented. Unlike many existing dec-POMDP-based RL approaches, the proposed algorithm is suitable for model-free multi-agent reinforcement learning as it does not require knowledge of a transition model. Furthermore, experimental results illustrate the efficacy of the algorithm and demonstrate its superiority over the current state-of-the-art methods.
Paper Structure (21 sections, 13 theorems, 118 equations, 3 figures, 1 algorithm)

This paper contains 21 sections, 13 theorems, 118 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. State of the art
  3. Settings
  4. State estimation in model-based learning
  5. State estimation in model-free RL
  6. Neural Network-based Approaches
  7. Non-neural-based Approach
  8. Consensus-based solutions
  9. Model Setting
  10. Multi-agent Off-Policy Actor-Critic for dec-POMDP
  11. Multi-agent off-policy actor-critic learning
  12. State estimation
  13. Importance sampling ratio
  14. Theoretical guarantees
  15. Experimental results and Discussions
  16. ...and 6 more sections

Key Result

Theorem 1

Let $\|\cdot\|$ denote the Euclidean norm ($2-$norm) of a vector. Then, under Assumptions assumption:combination-assumption:step, for all agents $k$, $\Delta \theta_{k,n}$ is $\varepsilon$-bounded at time $n$ if $\forall j \leq n$: where $\blacksquare$

Figures (3)

  • Figure 1: A block diagram illustrating the primary steps of the proposed algorithm.
  • Figure 2: Illustration of the agents/target framework: (a) depicts the initial state, corresponding to the position of a target (green) and the fixed positions of agents (blue); (b) shows a phase where agents exchange parameters according to the communication graph: black arrows demonstrate communication links; (c) illustrates a phase where agents, based on their individual policies, choose an action, i.e., select the possible location of the target (cells indicated by the orange arrows); (d) demonstrates the transition of the target to another state (cell) as a result of the agents' actions.
  • Figure 3: Comparison between MAOPAC and the proposed MAOPAC-dec-POMDP: (a) shows the difference in critic values computed by MAOPAC and MAOPAC-dec-POMDP for different numbers of agents: $K=3,K=5$ and $K=7$; (b) shows the difference in actor values computed by MAOPAC and MAOPAC-dec-POMDP for different numbers of agents: $K=5,K=7$ and $K=9$ (c) shows the network agreement for different numbers of agents: $K=3,K=5$ and $K=7$; (d) compares the proposed MAOPAC-dec-POMDP and ZOPO in terms of cumulative average reward

Theorems & Definitions (26)

  • Theorem 1: $\boldsymbol{\epsilon}$-optimality
  • Corollary 1: Boltzman distributions
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • ...and 16 more