Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Lead Critic based Multi-Agent Reinforcement Learning

David Eckel, Henri Meeß

TL;DR

A novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels, and a proposed Hierarchical Lead Critic (HLC), which demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies.

Abstract

Cooperative Multi-Agent Reinforcement Learning (MARL) solves complex tasks that require coordination from multiple agents, but is often limited to either local (independent learning) or global (centralized learning) perspectives. In this paper, we introduce a novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels. We propose the Hierarchical Lead Critic (HLC) - inspired by natural emerging distributions in team structures, where following high-level objectives combines with low-level execution. HLC demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies. Experimental results conducted on cooperative, non-communicative, and partially observable MARL benchmarks demonstrate that HLC outperforms single hierarchy baselines and scales robustly with increasing amounts of agents and difficulty.

Hierarchical Lead Critic based Multi-Agent Reinforcement Learning

TL;DR

A novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels, and a proposed Hierarchical Lead Critic (HLC), which demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies.

Abstract

Cooperative Multi-Agent Reinforcement Learning (MARL) solves complex tasks that require coordination from multiple agents, but is often limited to either local (independent learning) or global (centralized learning) perspectives. In this paper, we introduce a novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels. We propose the Hierarchical Lead Critic (HLC) - inspired by natural emerging distributions in team structures, where following high-level objectives combines with low-level execution. HLC demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies. Experimental results conducted on cooperative, non-communicative, and partially observable MARL benchmarks demonstrate that HLC outperforms single hierarchy baselines and scales robustly with increasing amounts of agents and difficulty.
Paper Structure (25 sections, 9 equations, 10 figures, 6 tables, 1 algorithm)

This paper contains 25 sections, 9 equations, 10 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. MARL
  4. Learning from Multiple Critics
  5. Preliminaries
  6. Hierarchical Lead Critic
  7. Lead Critics
  8. Sequential Updates
  9. HLC Actor
  10. HLC with SAC
  11. Experiments
  12. MOMAland Escort & Surveillance
  13. Experimental Results
  14. Ablation Study
  15. Conclusion
  16. ...and 10 more sections

Figures (10)

  • Figure 1: Hierarchical Lead Critic (HLC) structure. Agents are evaluated by all related critics (local critics and Lead Critics) sequentially.
  • Figure 2: HLC sequential updating scheme for a selected agent with a local critic and a Lead Critic.
  • Figure 3: HLC actor. The observation is processed through multiple paths that are combined through cross-attention and concatenation.
  • Figure 4: SimpleSpread \ref{['subfloat:SimpleSpread']}MPE_Pettingzoo, Momaland Escort \ref{['subfloat:Escort8']}, and Surveillance \ref{['subfloat:Surveillance4']} environments used in our evaluations. In SimpleSpread, black circles are landmarks and purple particles are agents. In Escort8, agent drones (red) escort a target (yellow), while keeping an implicit formation. In Surveillance4, agents observe a target with fixed relative altitude and distribute themselves evenly around the target.
  • Figure 5: Performance of HLC compared to HASAC and ISAC on the Escort tasks with episode lengths, Surveillance task and SimpleSpread.
  • ...and 5 more figures