Resolving Conflicting Constraints in Multi-Agent Reinforcement Learning with Layered Safety

TL;DR

This work tackles safety in dense multi-agent reinforcement learning by addressing conflicting pairwise safety constraints that arise in multi-robot navigation. It introduces Layered Safe MARL, a three-tier framework that combines MARL (to minimize interactions), a prioritization module (to identify urgent collision pairs), and a Control Barrier-Value Function (CBVF) safety filter (to enforce safety) grounded in Hamilton-Jacobi reachability. The training pipeline incorporates curriculum learning and safety-informed rewards to reduce conservatism, while ensuring safety via CBVF-based QP-based action corrections, including cooperative and non-cooperative filtering. The approach is validated through hardware experiments with Crazyflie drones and high-density AAM simulations (air taxis), showing significant reductions in conflicts and improved efficiency (faster travel times and fewer near-collisions) compared to baselines. The results demonstrate the viability and practicality of integrating model-based safety tools with learning-based strategies for scalable, safety-critical multi-agent coordination.

Abstract

Preventing collisions in multi-robot navigation is crucial for deployment. This requirement hinders the use of learning-based approaches, such as multi-agent reinforcement learning (MARL), on their own due to their lack of safety guarantees. Traditional control methods, such as reachability and control barrier functions, can provide rigorous safety guarantees when interactions are limited only to a small number of robots. However, conflicts between the constraints faced by different agents pose a challenge to safe multi-agent coordination. To overcome this challenge, we propose a method that integrates multiple layers of safety by combining MARL with safety filters. First, MARL is used to learn strategies that minimize multiple agent interactions, where multiple indicates more than two. Particularly, we focus on interactions likely to result in conflicting constraints within the engagement distance. Next, for agents that enter the engagement distance, we prioritize pairs requiring the most urgent corrective actions. Finally, a dedicated safety filter provides tactical corrective actions to resolve these conflicts. Crucially, the design decisions for all layers of this framework are grounded in reachability analysis and a control barrier-value function-based filtering mechanism. We validate our Layered Safe MARL framework in 1) hardware experiments using Crazyflie drones and 2) high-density advanced aerial mobility (AAM) operation scenarios, where agents navigate to designated waypoints while avoiding collisions. The results show that our method significantly reduces conflict while maintaining safety without sacrificing much efficiency (i.e., shorter travel time and distance) compared to baselines that do not incorporate layered safety. The project website is available at https://dinamo-mit.github.io/Layered-Safe-MARL/

Figures (8)

• Figure 1: The figure shows our approach using an example scenario of four agents. Agent $i$ must reach the waypoints shown on the right. Our Layered Safe MARL framework consists of three key components, and we describe it as applied through agent $i$: 1) The MARL policy generates an action based on the observation within the range $r_{\textrm{obs}}$ while aiming to reduce the likelihood of entering other agents' potential conflict range $r_{\textrm{conflict}}$. 2) The prioritization module identifies the most critical neighboring agent in a potential collision scenario by evaluating the CBVF. In this example, agent $j_1$ is within the potential conflict region and forms a potential collision pair. 3) The CBVF safety filter adjusts the action to ensure safe navigation.
• Figure 2: Running example illustrating the CBVF-based safe sets, safety filtering, and the leaky corner issue. (a) Visualization of the ego agent ($s^{(1)} = [0.4km, 0km, 0\degree, 110$$kt]$)'s maximal safe sets (exterior of the level sets) against two agents, $s^{(2)}\!=\![1.7 km, 0.3km, -120\degree, 110kt]$ and $s^{(3)} \!=\![1.7 km, -0.6km, -180\degree, 60kt]$. (b) In the two-agent case, each agent executing their CBVF safety filters \ref{['eq:cbvf-qp-single']} successfully prevents collision. (c) In the three-agent case, although agent 1 started inside the intersection of $\mathcal{S}^{(12)}$ and $\mathcal{S}^{(13)}$, it is not able to prevent safety violation. This is because the initial state of robot 1 is in the leaky corner.
• Figure 3: Maximum safe sets (exterior of the white level sets), potential conflict region, and CBVF (colormap) for each vehicle dynamics, displayed in the relative position space when (a) relative velocity is $(v_x, v_y) = (1, 1)$ [m/s], (b) relative speed and heading is 220 knots and $180$°, respectively.
• Figure 4: Crazyflie hardware experiment with the MARL policy learned by our method. The three drones have to pass through two common waypoints to get to their landing location. The trajectories corresponding to the video footage are visualized in Fig. \ref{['fig:hardware trajectory']} (b).
• Figure 5: We compare the recorded Crazyflie hardware experiment trajectories under our method and the baseline policy trained without the safety filter. With our approach, the drones smoothly deconflict and efficiently complete the task. In contrast, under the baseline policy, the yellow Crazyflie misses a waypoint and must make a second pass. These results demonstrate that incorporating layered safety information during training improves the performance of the MARL policy.

Theorems & Definitions (5)

• Remark 1
• Remark 2
• Proposition 1
• proof
• Remark 3