Enhancing Aerial Combat Tactics through Hierarchical Multi-Agent Reinforcement Learning

TL;DR

This work introduces a heterogeneous hierarchical MARL framework for analyzing simulated air combat by splitting decision-making into low-level controllers for individual aircraft and a high-level commander policy that guides mission-level actions. Training employs curriculum learning and fictitious self-play within a CTDE setting, with shared policies across identical aircraft types and temporally extended options that couple low-level control with strategic commands. The study demonstrates that carefully designed reward structures, attention-based and GRU-enabled architectures, and CTDE yield robust performance across varied combat scales, with the commander providing effective look-ahead planning and policy coordination. The findings suggest practical significance for defense simulations, offering scalable, explainable, and adaptable tactics generation in complex, heterogeneous multi-agent air combat scenarios.

Abstract

This work presents a Hierarchical Multi-Agent Reinforcement Learning framework for analyzing simulated air combat scenarios involving heterogeneous agents. The objective is to identify effective Courses of Action that lead to mission success within preset simulations, thereby enabling the exploration of real-world defense scenarios at low cost and in a safe-to-fail setting. Applying deep Reinforcement Learning in this context poses specific challenges, such as complex flight dynamics, the exponential size of the state and action spaces in multi-agent systems, and the capability to integrate real-time control of individual units with look-ahead planning. To address these challenges, the decision-making process is split into two levels of abstraction: low-level policies control individual units, while a high-level commander policy issues macro commands aligned with the overall mission targets. This hierarchical structure facilitates the training process by exploiting policy symmetries of individual agents and by separating control from command tasks. The low-level policies are trained for individual combat control in a curriculum of increasing complexity. The high-level commander is then trained on mission targets given pre-trained control policies. The empirical validation confirms the advantages of the proposed framework.

Figures (20)

• Figure 1: Aircraft attacking mechanism. Blue aircraft are (our) RL agents, red aircraft denote opponents. Shooting with a cannon is represented by a conical shape (WEZ), while firing a rocket is depicted with a corresponding symbol.
• Figure 2: MARL interaction cycle. $R_{i,t}$ denotes rewards, $o_{i,t}$ observations and $a_{i,t}$ are actions.
• Figure 3: Hierarchy of policies. The commander operates at the high-level dynamics for strategic planning by observing more information of the current situation, while fight and escape policy are deployed to control the aircraft at low-level dynamics. Each policy is equipped with an Actor for execution and a Critic for Learning.
• Figure 4: The main components representing our approach: (a) Within the environment, we differentiate between high- and low-level information, depending on which policy is being trained. Agents are equipped with either $\pi_f$ or $\pi_e$, and the commander decides which to activate. Opponent aircraft are not controlled by a commander. (b) Each aircraft type operates on its own network instance. The green layer is shared between both instances, as well as between their actors and critics.
• Figure 5: Aircraft metrics: (a) heading, (b) heading off, (c) aspect angle, (d) antenna train angle, (e) distance.