Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Safe Multi-Agent Reinforcement Learning for Multi-Agent System

Haikuo Du, Fandi Gou, Yunze Cai

TL;DR

SS-MARL addresses safety and scalability in multi-agent reinforcement learning by integrating a graph neural network backbone for implicit inter-agent communication with constrained joint policy optimization. The approach introduces a cost critic to enforce safety constraints alongside the reward-driven objective, and uses sequential, KL-constrained policy updates with a TRPO recovery step to maintain feasibility when multiple costs are present. Empirical results on modified MPE tasks show SS-MARL achieves a favorable balance between optimality and safety, and can zero-shot transfer from small to very large agent populations while maintaining high safety and performance. This combination of GNN-based communication and multi-constraint optimization significantly enhances practical applicability of MARL in large-scale, safety-critical MAS.

Abstract

Safety and scalability are two critical challenges faced by practical Multi-Agent Systems (MAS). However, existing Multi-Agent Reinforcement Learning (MARL) algorithms that rely solely on reward shaping are ineffective in ensuring safety, and their scalability is rather limited due to the fixed-size network output. To address these issues, we propose a novel framework, Scalable Safe MARL (SS-MARL), to enhance the safety and scalability of MARL methods. Leveraging the inherent graph structure of MAS, we design a multi-layer message passing network to aggregate local observations and communications of varying sizes. Furthermore, we develop a constrained joint policy optimization method in the setting of local observation to improve safety. Simulation experiments demonstrate that SS-MARL achieves a better trade-off between optimality and safety compared to baselines, and its scalability significantly outperforms the latest methods in scenarios with a large number of agents.

Scalable Safe Multi-Agent Reinforcement Learning for Multi-Agent System

TL;DR

SS-MARL addresses safety and scalability in multi-agent reinforcement learning by integrating a graph neural network backbone for implicit inter-agent communication with constrained joint policy optimization. The approach introduces a cost critic to enforce safety constraints alongside the reward-driven objective, and uses sequential, KL-constrained policy updates with a TRPO recovery step to maintain feasibility when multiple costs are present. Empirical results on modified MPE tasks show SS-MARL achieves a favorable balance between optimality and safety, and can zero-shot transfer from small to very large agent populations while maintaining high safety and performance. This combination of GNN-based communication and multi-constraint optimization significantly enhances practical applicability of MARL in large-scale, safety-critical MAS.

Abstract

Safety and scalability are two critical challenges faced by practical Multi-Agent Systems (MAS). However, existing Multi-Agent Reinforcement Learning (MARL) algorithms that rely solely on reward shaping are ineffective in ensuring safety, and their scalability is rather limited due to the fixed-size network output. To address these issues, we propose a novel framework, Scalable Safe MARL (SS-MARL), to enhance the safety and scalability of MARL methods. Leveraging the inherent graph structure of MAS, we design a multi-layer message passing network to aggregate local observations and communications of varying sizes. Furthermore, we develop a constrained joint policy optimization method in the setting of local observation to improve safety. Simulation experiments demonstrate that SS-MARL achieves a better trade-off between optimality and safety compared to baselines, and its scalability significantly outperforms the latest methods in scenarios with a large number of agents.
Paper Structure (14 sections, 9 equations, 6 figures, 2 tables)

This paper contains 14 sections, 9 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Safe MARL
  4. Partial Observability in MARL
  5. Scalability in MARL
  6. Problem Formulation
  7. Approach
  8. Actor, Critic and Cost Critic
  9. Constrained Joint Policy Optimization
  10. Experiments
  11. SS-MARL Safety Analysis
  12. Comparative Experiments
  13. Scalability Experiments
  14. Conclusion

Figures (6)

  • Figure 1: Overview of SS-MARL. AA: Agent Aggregation, GA: Graph Aggregation, GAE: Generalized Advantage Estimation.
  • Figure 2: Visual Representation of message passing and aggregations in GNN. (Note: The embedding layer during the initial message passing is not shown in the above figure)
  • Figure 3: Comparison of the training performance of SS-MARL with baselines. (a)(b)(c) are average rewards per step per agent during the training phase, (d)(e)(f) are average costs per step per agent during the training phase.
  • Figure 4: Scenarios with (a) $c=1$ and (b) $c=6$ and (c) average rewards per step per agent and average costs per step per agent during the training phase.
  • Figure 5: Zero-shot transfer to $n=24$ using a model trained on scenarios with $n=3$, when the test episode (left) begins and (right) ends
  • ...and 1 more figures