Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safe Heterogeneous Multi-Agent RL with Communication Regularization for Coordinated Target Acquisition

Gabriele Calzolari, Vidya Sumathy, Christoforos Kanellakis, George Nikolakopoulos

TL;DR

The paper tackles safe coordination of structurally heterogeneous robot teams (e.g., UAVs and UGVs) in unknown, partially observable environments with limited communication. It fuses MAPPO with a Graph Attention Network encoder operating on a dynamic neighbor graph defined by a radius $r_c$, and augments it with a trajectory-aware safety filter that rescales actions to avoid collisions; the safety filter selects a feasible action by evaluating trajectory feasibility over a horizon. It introduces a communication orthogonality regularizer to encourage diverse message embeddings and reduce redundancy. Comprehensive ablations and simulator experiments demonstrate that the approach yields safe, stable coordination and improved target acquisition performance, with balanced workload among heterogeneous agents. The work advances practical Safe MARL for heterogeneous teams and provides a path toward real-world deployment.

Abstract

This paper introduces a decentralized multi-agent reinforcement learning framework enabling structurally heterogeneous teams of agents to jointly discover and acquire randomly located targets in environments characterized by partial observability, communication constraints, and dynamic interactions. Each agent's policy is trained with the Multi-Agent Proximal Policy Optimization algorithm and employs a Graph Attention Network encoder that integrates simulated range-sensing data with communication embeddings exchanged among neighboring agents, enabling context-aware decision-making from both local sensing and relational information. In particular, this work introduces a unified framework that integrates graph-based communication and trajectory-aware safety through safety filters. The architecture is supported by a structured reward formulation designed to encourage effective target discovery and acquisition, collision avoidance, and de-correlation between the agents' communication vectors by promoting informational orthogonality. The effectiveness of the proposed reward function is demonstrated through a comprehensive ablation study. Moreover, simulation results demonstrate safe and stable task execution, confirming the framework's effectiveness.

Safe Heterogeneous Multi-Agent RL with Communication Regularization for Coordinated Target Acquisition

TL;DR

The paper tackles safe coordination of structurally heterogeneous robot teams (e.g., UAVs and UGVs) in unknown, partially observable environments with limited communication. It fuses MAPPO with a Graph Attention Network encoder operating on a dynamic neighbor graph defined by a radius , and augments it with a trajectory-aware safety filter that rescales actions to avoid collisions; the safety filter selects a feasible action by evaluating trajectory feasibility over a horizon. It introduces a communication orthogonality regularizer to encourage diverse message embeddings and reduce redundancy. Comprehensive ablations and simulator experiments demonstrate that the approach yields safe, stable coordination and improved target acquisition performance, with balanced workload among heterogeneous agents. The work advances practical Safe MARL for heterogeneous teams and provides a path toward real-world deployment.

Abstract

This paper introduces a decentralized multi-agent reinforcement learning framework enabling structurally heterogeneous teams of agents to jointly discover and acquire randomly located targets in environments characterized by partial observability, communication constraints, and dynamic interactions. Each agent's policy is trained with the Multi-Agent Proximal Policy Optimization algorithm and employs a Graph Attention Network encoder that integrates simulated range-sensing data with communication embeddings exchanged among neighboring agents, enabling context-aware decision-making from both local sensing and relational information. In particular, this work introduces a unified framework that integrates graph-based communication and trajectory-aware safety through safety filters. The architecture is supported by a structured reward formulation designed to encourage effective target discovery and acquisition, collision avoidance, and de-correlation between the agents' communication vectors by promoting informational orthogonality. The effectiveness of the proposed reward function is demonstrated through a comprehensive ablation study. Moreover, simulation results demonstrate safe and stable task execution, confirming the framework's effectiveness.
Paper Structure (15 sections, 18 equations, 4 figures, 1 table)

This paper contains 15 sections, 18 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related works
  3. Our contribution
  4. Methodology
  5. Reinforcement learning framework
  6. Policy and critic networks
  7. Safety filters
  8. Holonomic agents
  9. Differential drive agent
  10. Reward function formulation
  11. Results
  12. Training
  13. Training results
  14. Simulation results
  15. Conclusion

Figures (4)

  • Figure 1: Illustration of the VMAS environment in which three agents are trained to locate and acquire randomly placed targets (orange). The diagram highlights the key components of each agent’s policy, including the GNN encoder, MLP heads, and safety filters.
  • Figure 2: Comparison of the mean episode reward trajectories for the full multi-agent system (a), holonomic agents (b), and diff-drive agents (c) under different reward shaping configurations. Reward schemes are color-coded as follows: R1 (blue), R2 (orange), R3 (green), and R4 (grey).
  • Figure 3: (a) Entropy evolution for holonomic and diff-drive agents under the four reward configurations. (b) Mean number of acquired targets over time with standard deviation bands. (c) Probability mass function of the number of targets acquired at the final timestep. Colors denote reward schemes: R1 (blue), R2 (orange), R3 (green), and R4 (grey).
  • Figure 4: Per-agent target discovery over time for the four reward configurations (R1–R4). Each subplot shows the mean discovery rate across simulations, with colored curves indicating the diff-drive agent (green), holonomic agent 1 (orange), and holonomic agent 2 (purple).