Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learning

Wei Geng, Baidi Xiao, Rongpeng Li, Ning Wei, Dong Wang, Zhifeng Zhao

TL;DR

A novel decomposition-based multi-agent distributional RL method is proposed by approximating the globally shared noisy reward by a Gaussian Mixture Model and decomposing it into the combination of individual distributional local rewards, with which each agent can be updated locally through distributional RL.

Abstract

Generally, Reinforcement Learning (RL) agent updates its policy by repetitively interacting with the environment, contingent on the received rewards to observed states and undertaken actions. However, the environmental disturbance, commonly leading to noisy observations (e.g., rewards and states), could significantly shape the performance of agent. Furthermore, the learning performance of Multi-Agent Reinforcement Learning (MARL) is more susceptible to noise due to the interference among intelligent agents. Therefore, it becomes imperative to revolutionize the design of MARL, so as to capably ameliorate the annoying impact of noisy rewards. In this paper, we propose a novel decomposition-based multi-agent distributional RL method by approximating the globally shared noisy reward by a Gaussian mixture model (GMM) and decomposing it into the combination of individual distributional local rewards, with which each agent can be updated locally through distributional RL. Moreover, a diffusion model (DM) is leveraged for reward generation in order to mitigate the issue of costly interaction expenditure for learning distributions. Furthermore, the optimality of the distribution decomposition is theoretically validated, while the design of loss function is carefully calibrated to avoid the decomposition ambiguity. We also verify the effectiveness of the proposed method through extensive simulation experiments with noisy rewards. Besides, different risk-sensitive policies are evaluated in order to demonstrate the superiority of distributional RL in different MARL tasks.

Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learning

TL;DR

A novel decomposition-based multi-agent distributional RL method is proposed by approximating the globally shared noisy reward by a Gaussian Mixture Model and decomposing it into the combination of individual distributional local rewards, with which each agent can be updated locally through distributional RL.

Abstract

Generally, Reinforcement Learning (RL) agent updates its policy by repetitively interacting with the environment, contingent on the received rewards to observed states and undertaken actions. However, the environmental disturbance, commonly leading to noisy observations (e.g., rewards and states), could significantly shape the performance of agent. Furthermore, the learning performance of Multi-Agent Reinforcement Learning (MARL) is more susceptible to noise due to the interference among intelligent agents. Therefore, it becomes imperative to revolutionize the design of MARL, so as to capably ameliorate the annoying impact of noisy rewards. In this paper, we propose a novel decomposition-based multi-agent distributional RL method by approximating the globally shared noisy reward by a Gaussian mixture model (GMM) and decomposing it into the combination of individual distributional local rewards, with which each agent can be updated locally through distributional RL. Moreover, a diffusion model (DM) is leveraged for reward generation in order to mitigate the issue of costly interaction expenditure for learning distributions. Furthermore, the optimality of the distribution decomposition is theoretically validated, while the design of loss function is carefully calibrated to avoid the decomposition ambiguity. We also verify the effectiveness of the proposed method through extensive simulation experiments with noisy rewards. Besides, different risk-sensitive policies are evaluated in order to demonstrate the superiority of distributional RL in different MARL tasks.
Paper Structure (25 sections, 8 theorems, 50 equations, 8 figures, 8 tables, 2 algorithms)

This paper contains 25 sections, 8 theorems, 50 equations, 8 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background and Problem Formulation
  4. System Model
  5. Value Decomposition
  6. Distributional RL
  7. Problem Formulation
  8. Noise Distribution Decomposition for MARL
  9. GMM-based Distributional MARL
  10. Consistency of Global and Local Optimal Actions
  11. Design of Distortion Risk Function
  12. DM-based Data Augment
  13. Loss Function Design
  14. Experimental Settings and Simulation Results
  15. Experimental Settings
  16. ...and 10 more sections

Key Result

Theorem 1

For any $i=1,\dots,N$ and $Z_{i}$ defined by eq:local_z in any $m$-th step, if $w^{[i]}_m\geq 0$ we have

Figures (8)

  • Figure 1: The scenario and framework of NDD-based cooperative multi-agent distributional RL with noisy rewards.
  • Figure 2: Overview of the NDD algorithm.
  • Figure 3: Performance comparison between algorithms in (i) the MPE Reference task with Noise $0$; (ii) the SMAC 3m task with Noise $0$. Notably, the "Baseline" indicates the result of MAPPO under noise-free settings; "3m" means 3 Marines in MAS.
  • Figure 4: Performance sensitivity of NDD in Adversary with Noise $0$ under the settings of (i) different $\alpha$ with $\lambda=1$; (ii) different $\lambda$ with $\alpha=1$.
  • Figure 5: Results of distribution decomposition over two noise cases (i.e., $0.25\beta(1,2) + 0.75\mathcal{N}(-5,3)$, $0.35\mathcal{N}(-6,1)+0.3\beta(1,2)+0.35\chi(9)$). The left two sub-figures show the comparison between practical PDF histogram and weighted PDFs of decomposed distributions ($3$ colored curves); while the right sub-figure shows the curve of loss over training iterations. $\mathcal{N}, \beta, \mathcal{X}^2$ indicate a Gaussian, a Beta and a Chi-Square distribution, respectively.
  • ...and 3 more figures

Theorems & Definitions (14)

  • Theorem 1
  • proof
  • Corollary 1
  • Theorem 2
  • proof
  • Theorem 3: Bounded Generation Error
  • Theorem 4: Bounded Approximation Error
  • proof
  • Lemma 1
  • proof
  • ...and 4 more