Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration

Xudong Guo, Daming Shi, Junjie Yu, Wenhui Fan

TL;DR

SHPPO introduces a scalable, heterogeneous MARL framework that embeds agent-specific policy variance via a latent network and a per-agent heterogeneous layer, enabling zero-shot scalability to unseen team sizes. A centralized InferenceNet guides latent learning, while losses promote identifiable and diverse latent representations, yielding adaptive behavior without architecture blow-up. Evaluations on SMAC and GRF demonstrate competitive performance to heterogeneous baselines on original tasks and superior zero-shot transfer to varying populations, with insights into latent space correspondences to team strategies. This approach offers practical scalability for real-world multi-agent systems by decoupling heterogeneity from base parameter sharing and enabling flexible role assignment across dynamic team sizes.

Abstract

The emergence of multi-agent reinforcement learning (MARL) is significantly transforming various fields like autonomous vehicle networks. However, real-world multi-agent systems typically contain multiple roles, and the scale of these systems dynamically fluctuates. Consequently, in order to achieve zero-shot scalable collaboration, it is essential that strategies for different roles can be updated flexibly according to the scales, which is still a challenge for current MARL frameworks. To address this, we propose a novel MARL framework named Scalable and Heterogeneous Proximal Policy Optimization (SHPPO), integrating heterogeneity into parameter-shared PPO-based MARL networks. We first leverage a latent network to learn strategy patterns for each agent adaptively. Second, we introduce a heterogeneous layer to be inserted into decision-making networks, whose parameters are specifically generated by the learned latent variables. Our approach is scalable as all the parameters are shared except for the heterogeneous layer, and gains both inter-individual and temporal heterogeneity, allowing SHPPO to adapt effectively to varying scales. SHPPO exhibits superior performance in classic MARL environments like Starcraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF), showcasing enhanced zero-shot scalability, and offering insights into the learned latent variables' impact on team performance by visualization.

Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration

TL;DR

SHPPO introduces a scalable, heterogeneous MARL framework that embeds agent-specific policy variance via a latent network and a per-agent heterogeneous layer, enabling zero-shot scalability to unseen team sizes. A centralized InferenceNet guides latent learning, while losses promote identifiable and diverse latent representations, yielding adaptive behavior without architecture blow-up. Evaluations on SMAC and GRF demonstrate competitive performance to heterogeneous baselines on original tasks and superior zero-shot transfer to varying populations, with insights into latent space correspondences to team strategies. This approach offers practical scalability for real-world multi-agent systems by decoupling heterogeneity from base parameter sharing and enabling flexible role assignment across dynamic team sizes.

Abstract

The emergence of multi-agent reinforcement learning (MARL) is significantly transforming various fields like autonomous vehicle networks. However, real-world multi-agent systems typically contain multiple roles, and the scale of these systems dynamically fluctuates. Consequently, in order to achieve zero-shot scalable collaboration, it is essential that strategies for different roles can be updated flexibly according to the scales, which is still a challenge for current MARL frameworks. To address this, we propose a novel MARL framework named Scalable and Heterogeneous Proximal Policy Optimization (SHPPO), integrating heterogeneity into parameter-shared PPO-based MARL networks. We first leverage a latent network to learn strategy patterns for each agent adaptively. Second, we introduce a heterogeneous layer to be inserted into decision-making networks, whose parameters are specifically generated by the learned latent variables. Our approach is scalable as all the parameters are shared except for the heterogeneous layer, and gains both inter-individual and temporal heterogeneity, allowing SHPPO to adapt effectively to varying scales. SHPPO exhibits superior performance in classic MARL environments like Starcraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF), showcasing enhanced zero-shot scalability, and offering insights into the learned latent variables' impact on team performance by visualization.
Paper Structure (25 sections, 16 equations, 7 figures, 6 tables, 2 algorithms)

This paper contains 25 sections, 16 equations, 7 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. MARL for Collaboration
  4. Scaling MARL
  5. Heterogeneous MARL
  6. Background
  7. DEC-POMDP
  8. Policy Gradient (PG)
  9. PG in Multi-agent
  10. Scalable and Heterogeneous Multi-Agent Reinforcement Learning
  11. Scalable Latent Learning
  12. Heterogeneous Layer Design
  13. Loss Formulation
  14. Efficacy of the LatentNet and InferenceNet
  15. InferenceNet
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Illustration of scalability and heterogeneity in SMAC. (a) The original task without heterogeneous strategies, where different kinds of agents have the same strategy as a circle. (b) The original task with heterogeneous strategies, where the agents form two groups to attract the fire (blue) and attack in the distance (red). (c) The unseen task with a new agent (green) without heterogeneous strategies, where the new agent has the same strategy as others. (d) The new task with heterogeneous strategies, where the new agent can adaptively choose the strategy.
  • Figure 2: Schematics of our approach SHPPO. (a) Architecture of LatentNet to represent the agent's strategy pattern as a low-dimensional latent variable $l$. (b) Framework of SHPPO to conduct latent learning and action learning with dual actor-critic networks. Each agent contains two networks ActorNet and LatentNet. The two networks are parameter-shared among the agents except for the heterogeneous layer (HeteLayer), so SHPPO is scalable. (c) Architecture of ActorNet with HeteLayer, bringing heterogeneity in SHPPO. (d) Architecture of HeteLayer where the latent variable $l$ is decoded to the parameters of the heterogeneous linear layer. The pink blocks denote the losses. The yellow and green bars denote different HeteLayers.
  • Figure 3: Visualization of the tasks. (a) MMM2 in SMAC. (b) 8m_vs_9m in SMAC. (c) 3_vs_1_with_keeper in GRF. (d) counterattack _easy in GRF.
  • Figure 4: Performance on SMAC and GRF. We plot the win rate on SMAC and the score rate on GRF during training. (a) MMM2 (SMAC). (b) 8m_ vs_9m (SMAC). (c) 3_ vs_1_ with_ keeper (GRF). (d) counterattack_ easy (GRF). The confidence interval is calculated over 5 seeds.
  • Figure 5: Visualization of MMM2 and latent variables. The agents adaptively employ heterogeneous strategies throughout the task. The latent variables are clustered based on these strategies, indicated by circles of different colors: red: stay back to be covered. green: attack in the distance. blue: attract the fire by moving to the front. Each agent is identified by an ID, and their types are distinguished by the shapes in the lower figures: solid circles: Marauders, triangle: Marines, star: Medivac.
  • ...and 2 more figures