Embedded Mean Field Reinforcement Learning for Perimeter-defense Game

TL;DR

This work tackles large-scale, three-dimensional perimeter-defense under wind perturbations and heterogeneous agent dynamics by deriving Nash equilibrium strategies for one-on-one engagements and extending to many-versus-many settings via the Embedded Mean Field Actor-Critic (EMFAC) framework. EMFAC combines a high-level action encoder with mean-field aggregation and an agent-level reward-based attention mechanism to manage the exponential observation space and complex interactions, enabling scalable coordination in Dec-POMDP environments. Through extensive simulations and real-world Crazyflie experiments, EMFAC demonstrates faster convergence, higher accumulated reward, and lower collision rates across scales, validating its practical viability for complex defense tasks. The approach offers a principled, scalable solution for large-scale pursuit-evasion and multi-agent defense problems with realistic dynamics, wind effects, and heterogeneity.

Abstract

With the rapid advancement of unmanned aerial vehicles (UAVs) and missile technologies, perimeter-defense game between attackers and defenders for the protection of critical regions have become increasingly complex and strategically significant across a wide range of domains. However, existing studies predominantly focus on small-scale, simplified two-dimensional scenarios, often overlooking realistic environmental perturbations, motion dynamics, and inherent heterogeneity--factors that pose substantial challenges to real-world applicability. To bridge this gap, we investigate large-scale heterogeneous perimeter-defense game in a three-dimensional setting, incorporating realistic elements such as motion dynamics and wind fields. We derive the Nash equilibrium strategies for both attackers and defenders, characterize the victory regions, and validate our theoretical findings through extensive simulations. To tackle large-scale heterogeneous control challenges in defense strategies, we propose an Embedded Mean-Field Actor-Critic (EMFAC) framework. EMFAC leverages representation learning to enable high-level action aggregation in a mean-field manner, supporting scalable coordination among defenders. Furthermore, we introduce a lightweight agent-level attention mechanism based on reward representation, which selectively filters observations and mean-field information to enhance decision-making efficiency and accelerate convergence in large-scale tasks. Extensive simulations across varying scales demonstrate the effectiveness and adaptability of EMFAC, which outperforms established baselines in both convergence speed and overall performance. To further validate practicality, we test EMFAC in small-scale real-world experiments and conduct detailed analyses, offering deeper insights into the framework's effectiveness in complex scenarios.

Figures (12)

• Figure 1: Hemispherical defense scenario with kinematic agents. Attacker $A$ (red) attempts to breach perimeter $\partial\mathcal{H}_R$ while defender $D$ (blue) intercepts.
• Figure 2: (a) The Transformed 2D Plane. (b) Proportional distances maintained through similarity.
• Figure 3: Framework of the EMFAC Method. The left part shows the training of the high-level action encoder $E_a$ and attention module $f_{\text{att}}$, while the right illustrates the mean-field RL process using them. Both modules are trained alternately.
• Figure 4: Simulation results for Nash equilibrium validation. (a) The time required for both attackers and defenders to reach the breach point and their corresponding payoffs. The three scenarios, from left to right, are: (i) both the attacker and defender adhering to the Nash equilibrium strategy, (ii) only the attacker following the Nash equilibrium strategy, and (iii) only the defender adopting the Nash equilibrium strategy. (b) Visualization of the decision perimeter that delineates the winning regions for both sides, with the initial defender position set at $[0.2, 0.2, 0.0]$.
• Figure 5: Comparison of average reward learning curves for different algorithms in perimeter-defense game tasks of varying scales. (a)Success Rate. (b) Collision Rate.

Theorems & Definitions (8)

• Theorem 1: Nash Equilibrium Strategy
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Theorem 4: Zero-Payoff Surface
• proof