Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformer-based Multi-agent Reinforcement Learning for Separation Assurance in Structured and Unstructured Airspaces

Arsyi Aziz, Peng Wei

TL;DR

This work tackles the challenge of aircraft separation assurance in diverse airspaces for Advanced Air Mobility by formulating a multi-agent reinforcement learning solution that leverages a transformer encoder operating on a relative polar state representation. A classifier token conditioned on ownship information aggregates intruder tokens, enabling a scalable, variable-count intruder handling mechanism and providing speed advisories to keep aircraft near their desired speeds. The model is trained with varied, procedurally generated sector structures and reaction policies (PPO) and evaluated in both structured and unstructured airspaces, demonstrating strong generalization, near-zero near mid-air collision rates, and improved safety compared to a pure attention baseline. These results indicate that the combination of a polar state formulation, token-based transformer architecture, and diverse training scenarios yields a flexible, decentralized approach to separation assurance with potential for deployment in future high-density AAM operations.

Abstract

Conventional optimization-based metering depends on strict adherence to precomputed schedules, which limits the flexibility required for the stochastic operations of Advanced Air Mobility (AAM). In contrast, multi-agent reinforcement learning (MARL) offers a decentralized, adaptive framework that can better handle uncertainty, required for safe aircraft separation assurance. Despite this advantage, current MARL approaches often overfit to specific airspace structures, limiting their adaptability to new configurations. To improve generalization, we recast the MARL problem in a relative polar state space and train a transformer encoder model across diverse traffic patterns and intersection angles. The learned model provides speed advisories to resolve conflicts while maintaining aircraft near their desired cruising speeds. In our experiments, we evaluated encoder depths of 1, 2, and 3 layers in both structured and unstructured airspaces, and found that a single encoder configuration outperformed deeper variants, yielding near-zero near mid-air collision rates and shorter loss-of-separation infringements than the deeper configurations. Additionally, we showed that the same configuration outperforms a baseline model designed purely with attention. Together, our results suggest that the newly formulated state representation, novel design of neural network architecture, and proposed training strategy provide an adaptable and scalable decentralized solution for aircraft separation assurance in both structured and unstructured airspaces.

Transformer-based Multi-agent Reinforcement Learning for Separation Assurance in Structured and Unstructured Airspaces

TL;DR

This work tackles the challenge of aircraft separation assurance in diverse airspaces for Advanced Air Mobility by formulating a multi-agent reinforcement learning solution that leverages a transformer encoder operating on a relative polar state representation. A classifier token conditioned on ownship information aggregates intruder tokens, enabling a scalable, variable-count intruder handling mechanism and providing speed advisories to keep aircraft near their desired speeds. The model is trained with varied, procedurally generated sector structures and reaction policies (PPO) and evaluated in both structured and unstructured airspaces, demonstrating strong generalization, near-zero near mid-air collision rates, and improved safety compared to a pure attention baseline. These results indicate that the combination of a polar state formulation, token-based transformer architecture, and diverse training scenarios yields a flexible, decentralized approach to separation assurance with potential for deployment in future high-density AAM operations.

Abstract

Conventional optimization-based metering depends on strict adherence to precomputed schedules, which limits the flexibility required for the stochastic operations of Advanced Air Mobility (AAM). In contrast, multi-agent reinforcement learning (MARL) offers a decentralized, adaptive framework that can better handle uncertainty, required for safe aircraft separation assurance. Despite this advantage, current MARL approaches often overfit to specific airspace structures, limiting their adaptability to new configurations. To improve generalization, we recast the MARL problem in a relative polar state space and train a transformer encoder model across diverse traffic patterns and intersection angles. The learned model provides speed advisories to resolve conflicts while maintaining aircraft near their desired cruising speeds. In our experiments, we evaluated encoder depths of 1, 2, and 3 layers in both structured and unstructured airspaces, and found that a single encoder configuration outperformed deeper variants, yielding near-zero near mid-air collision rates and shorter loss-of-separation infringements than the deeper configurations. Additionally, we showed that the same configuration outperforms a baseline model designed purely with attention. Together, our results suggest that the newly formulated state representation, novel design of neural network architecture, and proposed training strategy provide an adaptable and scalable decentralized solution for aircraft separation assurance in both structured and unstructured airspaces.
Paper Structure (25 sections, 10 equations, 4 figures, 4 tables)

This paper contains 25 sections, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transformer Networks
  4. Multi-agent Reinforcement Learning for Aircraft Separation Assurance
  5. Problem Formulation
  6. Safety-Critical Events
  7. Conflict
  8. Loss of Separation
  9. Near Mid-Air Collision
  10. Mathematical Formulation
  11. State Space
  12. Action Space
  13. Reward Function
  14. Neural Network Architecture
  15. Model Training
  16. ...and 10 more sections

Figures (4)

  • Figure 1: The implemented network adopts an encoder–transformer architecture, where intruder information is represented as intruder tokens. A classifier ($\texttt{[CLS]}$) token, derived from ownship information, is appended to this set. The complete token sequence is processed by a transformer encoder with $M$ layers. The final-layer $\texttt{[CLS]}$ token is used to produce both the policy and the state-value estimates.
  • Figure 2: The simulated training environment consists of two routes whose endpoints are uniformly sampled within a circle of radius $30$ nautical miles. The routes may intersect at angles $\theta \in [0^\circ, 180^\circ]$ and can be traversed in either direction.
  • Figure 3: Evaluation of the model under three scenarios simulating structured and unstructured airspace. Case (\ref{['subfig:case1']}) and Case (\ref{['subfig:case2']}) represent structured airspace, but with rotational variability. Case (\ref{['subfig:case3']}) serves as a proxy for unstructured airspace, which includes one intermediary waypoint. For visualization purposes, we only show two routes in Case (\ref{['subfig:case3']}). However, the possible number of routes is dictated by the spawning scheme defined in Table \ref{['tab:spawning-details']}.
  • Figure 4: The average $\lambda$-returns (\ref{['fig:training-return']}) and entropy (\ref{['fig:training-entropy']}) of the transformer network during training, shown for configurations with $1$, $2$, and $3$ encoder layers. Each configuration was trained using three random seeds. Solid lines represent the mean values, while shaded regions indicate the range between the minimum and maximum values. Lines are smoothed with an exponential moving average with $\alpha=0.05$.