Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prioritized League Reinforcement Learning for Large-Scale Heterogeneous Multiagent Systems

Qingxu Fu, Zhiqiang Pu, Min Chen, Tenghai Qiu, Jianqiang Yi

TL;DR

A Prioritized Heterogeneous League Reinforcement Learning (PHLRL) method to address large-scale heterogeneous cooperation problems and uses Unreal Engine to design a large-scale heterogeneous cooperation benchmark named Large-Scale Multiagent Operation (LSMO), which is a complex two-team competition scenario that requires collaboration from both ground and airborne agents.

Abstract

Large-scale heterogeneous multiagent systems feature various realistic factors in the real world, such as agents with diverse abilities and overall system cost. In comparison to homogeneous systems, heterogeneous systems offer significant practical advantages. Nonetheless, they also present challenges for multiagent reinforcement learning, including addressing the non-stationary problem and managing an imbalanced number of agents with different types. We propose a Prioritized Heterogeneous League Reinforcement Learning (PHLRL) method to address large-scale heterogeneous cooperation problems. PHLRL maintains a record of various policies that agents have explored during their training and establishes a heterogeneous league consisting of diverse policies to aid in future policy optimization. Furthermore, we design a prioritized policy gradient approach to compensate for the gap caused by differences in the number of different types of agents. Next, we use Unreal Engine to design a large-scale heterogeneous cooperation benchmark named Large-Scale Multiagent Operation (LSMO), which is a complex two-team competition scenario that requires collaboration from both ground and airborne agents. We use experiments to show that PHLRL outperforms state-of-the-art methods, including QTRAN and QPLEX in LSMO.

Prioritized League Reinforcement Learning for Large-Scale Heterogeneous Multiagent Systems

TL;DR

A Prioritized Heterogeneous League Reinforcement Learning (PHLRL) method to address large-scale heterogeneous cooperation problems and uses Unreal Engine to design a large-scale heterogeneous cooperation benchmark named Large-Scale Multiagent Operation (LSMO), which is a complex two-team competition scenario that requires collaboration from both ground and airborne agents.

Abstract

Large-scale heterogeneous multiagent systems feature various realistic factors in the real world, such as agents with diverse abilities and overall system cost. In comparison to homogeneous systems, heterogeneous systems offer significant practical advantages. Nonetheless, they also present challenges for multiagent reinforcement learning, including addressing the non-stationary problem and managing an imbalanced number of agents with different types. We propose a Prioritized Heterogeneous League Reinforcement Learning (PHLRL) method to address large-scale heterogeneous cooperation problems. PHLRL maintains a record of various policies that agents have explored during their training and establishes a heterogeneous league consisting of diverse policies to aid in future policy optimization. Furthermore, we design a prioritized policy gradient approach to compensate for the gap caused by differences in the number of different types of agents. Next, we use Unreal Engine to design a large-scale heterogeneous cooperation benchmark named Large-Scale Multiagent Operation (LSMO), which is a complex two-team competition scenario that requires collaboration from both ground and airborne agents. We use experiments to show that PHLRL outperforms state-of-the-art methods, including QTRAN and QPLEX in LSMO.
Paper Structure (19 sections, 16 equations, 7 figures, 2 tables)

This paper contains 19 sections, 16 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Heterogeneous Multiagent System.
  5. League Training.
  6. Prioritized Heterogeneous League Reinforcement Learning
  7. Heterogeneous League.
  8. Learning Robust Heterogeneous Policies.
  9. Prioritized Policy Gradient.
  10. League Update
  11. Adaptive Hypernetwork-Based Neural Policy
  12. Experiments
  13. Heterogeneous Benchmark Environment.
  14. Experimental Setup.
  15. Overall Performance of PHLRL
  16. ...and 4 more sections

Figures (7)

  • Figure 1: A high-level illustration of the training procedure of PHLRL method. In each iteration, the frontier agent policies are mixed with historical league policies. After agents explore the large-scale heterogeneous environment, the sample of each episodes are prioritized according to the relative performance of each agent type. Finally, the frontier agent policies are optimized with the prioritized policy gradient calculated from these samples.
  • Figure 2: The policy and critic neural network structure of PHLRL method. The key difference with regular agent policies is the application of hyper networks.
  • Figure 3: The Large Scale multiagent OPeration (LSOP) benchmark environment.
  • Figure 4: Detour heterogeneous cooperative policy learned by PHLRL in LSOP. PHLRL agents are in green color, while their opponents are in blue color.
  • Figure 5: Direct confrontation and flanking policy learned by PHLRL in LSOP. PHLRL agents are in green color, while their opponents are in blue color.
  • ...and 2 more figures