Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed Multi-Agent Reinforcement Learning with One-hop Neighbors and Compute Straggler Mitigation

Baoqian Wang, Junfei Xie, Nikolay Atanasov

TL;DR

This work tackles the scalability of multi-agent reinforcement learning by introducing DARL1N, which factorizes value and policy representations to depend only on one-hop neighbors within a proximity graph, enabling distributed, off-policy actor-critic learning. It further advances robustness to compute stragglers with a Coded DARL1N architecture that encodes redundant computations across multiple learners and provides unbiased gradient estimates, converging similarly to SGD-based methods. The authors analyze several assignment schemes (MDS, Random Sparse, Repetition, LDGM) in terms of computation overhead and straggler resilience, and validate the approach through extensive experiments across five MARL benchmarks, showing faster training and competitive or superior policy quality, especially as the agent count grows. The combined algorithmic and coding-architecture contributions offer a practical path to scalable, resilient MARL in distributed settings, with guidance on selecting coding schemes based on system stability and performance priorities.

Abstract

Most multi-agent reinforcement learning (MARL) methods are limited in the scale of problems they can handle. With increasing numbers of agents, the number of training iterations required to find the optimal behaviors increases exponentially due to the exponentially growing joint state and action spaces. This paper tackles this limitation by introducing a scalable MARL method called Distributed multi-Agent Reinforcement Learning with One-hop Neighbors (DARL1N). DARL1N is an off-policy actor-critic method that addresses the curse of dimensionality by restricting information exchanges among the agents to one-hop neighbors when representing value and policy functions. Each agent optimizes its value and policy functions over a one-hop neighborhood, significantly reducing the learning complexity, yet maintaining expressiveness by training with varying neighbor numbers and states. This structure allows us to formulate a distributed learning framework to further speed up the training procedure. Distributed computing systems, however, contain straggler compute nodes, which are slow or unresponsive due to communication bottlenecks, software or hardware problems. To mitigate the detrimental straggler effect, we introduce a novel coded distributed learning architecture, which leverages coding theory to improve the resilience of the learning system to stragglers. Comprehensive experiments show that DARL1N significantly reduces training time without sacrificing policy quality and is scalable as the number of agents increases. Moreover, the coded distributed learning architecture improves training efficiency in the presence of stragglers.

Distributed Multi-Agent Reinforcement Learning with One-hop Neighbors and Compute Straggler Mitigation

TL;DR

This work tackles the scalability of multi-agent reinforcement learning by introducing DARL1N, which factorizes value and policy representations to depend only on one-hop neighbors within a proximity graph, enabling distributed, off-policy actor-critic learning. It further advances robustness to compute stragglers with a Coded DARL1N architecture that encodes redundant computations across multiple learners and provides unbiased gradient estimates, converging similarly to SGD-based methods. The authors analyze several assignment schemes (MDS, Random Sparse, Repetition, LDGM) in terms of computation overhead and straggler resilience, and validate the approach through extensive experiments across five MARL benchmarks, showing faster training and competitive or superior policy quality, especially as the agent count grows. The combined algorithmic and coding-architecture contributions offer a practical path to scalable, resilient MARL in distributed settings, with guidance on selecting coding schemes based on system stability and performance priorities.

Abstract

Most multi-agent reinforcement learning (MARL) methods are limited in the scale of problems they can handle. With increasing numbers of agents, the number of training iterations required to find the optimal behaviors increases exponentially due to the exponentially growing joint state and action spaces. This paper tackles this limitation by introducing a scalable MARL method called Distributed multi-Agent Reinforcement Learning with One-hop Neighbors (DARL1N). DARL1N is an off-policy actor-critic method that addresses the curse of dimensionality by restricting information exchanges among the agents to one-hop neighbors when representing value and policy functions. Each agent optimizes its value and policy functions over a one-hop neighborhood, significantly reducing the learning complexity, yet maintaining expressiveness by training with varying neighbor numbers and states. This structure allows us to formulate a distributed learning framework to further speed up the training procedure. Distributed computing systems, however, contain straggler compute nodes, which are slow or unresponsive due to communication bottlenecks, software or hardware problems. To mitigate the detrimental straggler effect, we introduce a novel coded distributed learning architecture, which leverages coding theory to improve the resilience of the learning system to stragglers. Comprehensive experiments show that DARL1N significantly reduces training time without sacrificing policy quality and is scalable as the number of agents increases. Moreover, the coded distributed learning architecture improves training efficiency in the presence of stragglers.
Paper Structure (40 sections, 6 theorems, 19 equations, 7 figures, 6 tables, 1 algorithm)

This paper contains 40 sections, 6 theorems, 19 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Distributed multi-Agent Reinforcement Learning with One-hop Neighbors (DARL1N)
  4. Coded Distributed Learning Architecture
  5. Assignment Matrix Construction and Assessment
  6. Assignment Matrix Construction
  7. Uncoded Assignment Scheme
  8. Coded Assignment Schemes
  9. Analysis and Comparison of Assignment Schemes
  10. Computation Overhead
  11. Resilience to Stragglers
  12. Experiments
  13. Performance of DARL1N
  14. Experiment Settings
  15. Environment Configurations
  16. ...and 25 more sections

Key Result

Lemma 1

If the absolute value of agent $i$'s reward is upper bounded as $|r_i(\mathbf{s}_{{\@fontswitch\mathcal{N}}_i},\mathbf{a}_{{\@fontswitch\mathcal{N}}_i})|\leq\bar{r}$, for some $\bar{r}>0$, the approximation error between $\tilde{Q}_i^{\boldsymbol{\mu}}(\mathbf{s}_{{\@fontswitch\mathcal{N}}_i}, \math

Figures (7)

  • Figure 1: (a) One-hop neighbor transitions from one time step to the next in a $d$-disk proximity graph; (b) Coded distributed learning architecture.
  • Figure 2: Average training time of different methods to run (a) 10 iterations in the Ising Model, (b) 30 iterations in the Food Collection, (c) 30 iterations in the Grassland, and (d) 30 iterations in the Adversarial Battle environments.
  • Figure 3: Average total training reward of different methods in the Food Collection environment when there are (a) $M=12$, and (b) $M=24$ agents.
  • Figure 4: (a) Average total training reward of different methods in the Food Collection environment with global reward and transition models when there are $M=8$ agents and (b) Mean and standard deviation of normalized total reward of competing agents trained by different methods in the Adversarial Battle environment with $M=48$.
  • Figure 5: Average total training reward of DARL1N and SAC in the Multi-Access Wireless Communication environment when (a) $z=2$ (b) $z=10$.
  • ...and 2 more figures

Theorems & Definitions (9)

  • Lemma 1
  • proof
  • Proposition 1
  • proof
  • Theorem 1
  • proof
  • Proposition 2
  • Proposition 3
  • Proposition 4