A Hierarchical Reinforcement Learning Framework for Multi-UAV Combat Using Leader-Follower Strategy

TL;DR

This work tackles the challenge of cooperative decision-making in large-scale multi-UAV air combat by introducing a three-level hierarchical framework driven by a Leader-Follower Multi-Agent Proximal Policy Optimization (LFMAPPO) scheme. The top-level policy selector assigns sub-policies, the middle level outputs desired action angles, and the bottom level translates these into precise 6-DOF commands, while a Target Selector assesses threat levels using situational context. By differentiating the value functions for leaders and followers and employing a max-min training objective, the approach fosters coordinated strategies that outperform baselines in both reward and win-rate metrics in simulated 6v6 scenarios. The combination of HRL, leader-follower roles, and a dedicated target-scoring mechanism offers a scalable path toward robust cooperative autonomy in complex, high-dimensional aerial combat. These advances have practical implications for autonomous air-defense and adversarial training, enabling more efficient and resilient multi-UAV operations.

Abstract

Multi-UAV air combat is a complex task involving multiple autonomous UAVs, an evolving field in both aerospace and artificial intelligence. This paper aims to enhance adversarial performance through collaborative strategies. Previous approaches predominantly discretize the action space into predefined actions, limiting UAV maneuverability and complex strategy implementation. Others simplify the problem to 1v1 combat, neglecting the cooperative dynamics among multiple UAVs. To address the high-dimensional challenges inherent in six-degree-of-freedom space and improve cooperation, we propose a hierarchical framework utilizing the Leader-Follower Multi-Agent Proximal Policy Optimization (LFMAPPO) strategy. Specifically, the framework is structured into three levels. The top level conducts a macro-level assessment of the environment and guides execution policy. The middle level determines the angle of the desired action. The bottom level generates precise action commands for the high-dimensional action space. Moreover, we optimize the state-value functions by assigning distinct roles with the leader-follower strategy to train the top-level policy, followers estimate the leader's utility, promoting effective cooperation among agents. Additionally, the incorporation of a target selector, aligned with the UAVs' posture, assesses the threat level of targets. Finally, simulation experiments validate the effectiveness of our proposed method.

Figures (7)

• Figure 1: Three different scenarios of UAV combat. (a) Single-UAV combat: An individual UAV engages in combat against a single opponent. (b) Multi-UAV in one-on-one combat: Each UAV is paired with a corresponding adversary, emphasizing independent engagements without coordination between UAVs. (c) Cooperative multi-UAV combat: A coordinated engagement where multiple UAVs work together as a team against a group of adversaries.
• Figure 2: Schematic diagram of multi-UAV combat mission. The UAVs are divided into two teams. The superscripts S and E represent the starting and ending positions. The dashed lines show the relative distances between the UAVs, and the vector arrows indicate their velocity directions. Initially, $A_2$ (follower) is positioned within the weapon engagement zone (WEZ) dillon2023optimal of $B_1$, represented by the sector-shaped area. $A_2$ accelerates to escape from $B_1$'s WEZ, and as $B_1$ pursues, it enters the WEZ of $A_1$ (leader).
• Figure 3: Overview of the Proposed Hierarchical Leader-Follower control framework. The top level, a policy selector oversees macro-level decision-making, assigning appropriate sub-policies based on different UAV tactical strategies. The middle level generates desired action adjustments in response to the current state, while the bottom level translates these into specific action commands. A target selection module provides the middle level with target information to enhance decision-making efficiency.
• Figure 4: The effectiveness of the algorithm is analyzed for two perspectives, average reward and win rate. (a) Average return curve: The average reward values of different methods during UAV training tasks over training episodes. As training progresses, the performance of each method improves gradually. Our approach demonstrate superior performance in the task, achieving higher reward values. (b) Win-Draw-Loss trends: The trends in win rates, draw rates, and loss rates of UAVs across different episodes. The win rate gradually increases, while the draw and loss rates decline correspondingly, reflecting the progressive improvement of the model.
• Figure 5: A comprehensive comparative analysis of algorithms in the air combat environment. It shows respective win rates after training for 50 episodes (a) and 100 episodes (b). please check grammar