Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention

Min Yang, Guanjun Liu, Ziyuan Zhou

TL;DR

Experimental results show that GPMF outperforms baselines including state-of-the-art partially observable mean field reinforcement learning algorithms, providing technical support for large-scale multi-UAV coordination and confrontation tasks in communication-constrained environments.

Abstract

Traditional multi-agent reinforcement learning algorithms are difficultly applied in a large-scale multi-agent environment. The introduction of mean field theory has enhanced the scalability of multi-agent reinforcement learning in recent years. This paper considers partially observable multi-agent reinforcement learning (MARL), where each agent can only observe other agents within a fixed range. This partial observability affects the agent's ability to assess the quality of the actions of surrounding agents. This paper focuses on developing a method to capture more effective information from local observations in order to select more effective actions. Previous work in this field employs probability distributions or weighted mean field to update the average actions of neighborhood agents, but it does not fully consider the feature information of surrounding neighbors and leads to a local optimum. In this paper, we propose a novel multi-agent reinforcement learning algorithm, Partially Observable Mean Field Multi-Agent Reinforcement Learning based on Graph-Attention (GAMFQ) to remedy this flaw. GAMFQ uses a graph attention module and a mean field module to describe how an agent is influenced by the actions of other agents at each time step. This graph attention module consists of a graph attention encoder and a differentiable attention mechanism, and this mechanism outputs a dynamic graph to represent the effectiveness of neighborhood agents against central agents. The mean-field module approximates the effect of a neighborhood agent on a central agent as the average effect of effective neighborhood agents. Experiments show that GAMFQ outperforms baselines including the state-of-the-art partially observable mean-field reinforcement learning algorithms. The code for this paper is here \url{https://github.com/yangmin32/GPMF}.

Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention

TL;DR

Experimental results show that GPMF outperforms baselines including state-of-the-art partially observable mean field reinforcement learning algorithms, providing technical support for large-scale multi-UAV coordination and confrontation tasks in communication-constrained environments.

Abstract

Traditional multi-agent reinforcement learning algorithms are difficultly applied in a large-scale multi-agent environment. The introduction of mean field theory has enhanced the scalability of multi-agent reinforcement learning in recent years. This paper considers partially observable multi-agent reinforcement learning (MARL), where each agent can only observe other agents within a fixed range. This partial observability affects the agent's ability to assess the quality of the actions of surrounding agents. This paper focuses on developing a method to capture more effective information from local observations in order to select more effective actions. Previous work in this field employs probability distributions or weighted mean field to update the average actions of neighborhood agents, but it does not fully consider the feature information of surrounding neighbors and leads to a local optimum. In this paper, we propose a novel multi-agent reinforcement learning algorithm, Partially Observable Mean Field Multi-Agent Reinforcement Learning based on Graph-Attention (GAMFQ) to remedy this flaw. GAMFQ uses a graph attention module and a mean field module to describe how an agent is influenced by the actions of other agents at each time step. This graph attention module consists of a graph attention encoder and a differentiable attention mechanism, and this mechanism outputs a dynamic graph to represent the effectiveness of neighborhood agents against central agents. The mean-field module approximates the effect of a neighborhood agent on a central agent as the average effect of effective neighborhood agents. Experiments show that GAMFQ outperforms baselines including the state-of-the-art partially observable mean-field reinforcement learning algorithms. The code for this paper is here \url{https://github.com/yangmin32/GPMF}.
Paper Structure (22 sections, 5 theorems, 25 equations, 8 figures, 4 tables, 1 algorithm)

This paper contains 22 sections, 5 theorems, 25 equations, 8 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Motivation & Preliminaries
  4. Partially observable Markov decision process
  5. Partially Observable Mean Field Reinforcement Learning
  6. Graph Attention Network
  7. Approach
  8. Graph--Attention Module
  9. Mean Field Module
  10. Theoretical Proof
  11. Algorithm
  12. Experiments
  13. Environments and Tasks
  14. Evaluation
  15. Hyperparameters
  16. ...and 7 more sections

Key Result

lemma 1

Srirampomfrl2021 When the Q-function is updated using the partially observable update rule in Eq.eq2, and assumptions assumption3, assumption5, and assumption7 hold, the following holds for $t \to \infty$: where $Q_*$ is the Nash Q-value, $Q_{POMF}$ is the partially observable mean-field Q-function, and $D$ is the bound of the $F$ map. The probability that the above formula holds is at least $\de

Figures (8)

  • Figure 1: A battle environment of the red and blue groups, where the red agent in the center is distributed by Dirichlet to calculate the action.
  • Figure 2: Schematic of GAMFQ. Each agent can observe the feature information of other agents within a fixed range, input it into the Graph--Attention Module, and output an adjacency matrix to represent the effectiveness of the neighborhood agent to the central agent.
  • Figure 3: Train results of three games. The reward curve for each algorithm is fitted by the least squares method.
  • Figure 4: Faceoff results of three games. The * in the legend indicates the enemy. For example, the first blue bar in the bar graph corresponding to the GAMFQ algorithm is the result of the confrontation between GAMFQ and MFQ, and we do not conduct confrontation experiments between the same algorithms.
  • Figure 5: Visualization of the standoff between GAMFQ and POMFQ (FOR) in a Multibattle game.
  • ...and 3 more figures

Theorems & Definitions (5)

  • lemma 1
  • theorem 1
  • theorem 2
  • theorem 3
  • theorem 4