Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Action-Stacking Improves Reinforcement Learning Performance for Air Traffic Control

Ben Carvell, George De Ath, Eseoghene Benjamin, Richard Everson

TL;DR

The paper addresses reinforcement learning for air traffic control under operational constraints, introducing online action-stacking as an inference-time wrapper that compiles bursts of primitive actions into domain-appropriate macro-clearances while training on a small action set. Using PPO on the BluebirdDT platform, the authors show that a five-action policy with action-damping and stacking can match or exceed the performance of a policy trained with a 37-action space in lateral navigation, and extend the approach to vertical control and two-aircraft avoidance. The key contributions include a practical mechanism to reduce instruction frequency, maintain safety, and scale RL to complex ATC tasks without changing the underlying MDP, enabling more realistic and tractable training. This technique offers a scalable pathway to operational ATC automation by bridging standard RL formulations with the realities of controller workflows and safety requirements.

Abstract

We introduce online action-stacking, an inference-time wrapper for reinforcement learning policies that produces realistic air traffic control commands while allowing training on a much smaller discrete action space. Policies are trained with simple incremental heading or level adjustments, together with an action-damping penalty that reduces instruction frequency and leads agents to issue commands in short bursts. At inference, online action-stacking compiles these bursts of primitive actions into domain-appropriate compound clearances. Using Proximal Policy Optimisation and the BluebirdDT digital twin platform, we train agents to navigate aircraft along lateral routes, manage climb and descent to target flight levels, and perform two-aircraft collision avoidance under a minimum separation constraint. In our lateral navigation experiments, action stacking greatly reduces the number of issued instructions relative to a damped baseline and achieves comparable performance to a policy trained with a 37-dimensional action space, despite operating with only five actions. These results indicate that online action-stacking helps bridge a key gap between standard reinforcement learning formulations and operational ATC requirements, and provides a simple mechanism for scaling to more complex control scenarios.

Online Action-Stacking Improves Reinforcement Learning Performance for Air Traffic Control

TL;DR

The paper addresses reinforcement learning for air traffic control under operational constraints, introducing online action-stacking as an inference-time wrapper that compiles bursts of primitive actions into domain-appropriate macro-clearances while training on a small action set. Using PPO on the BluebirdDT platform, the authors show that a five-action policy with action-damping and stacking can match or exceed the performance of a policy trained with a 37-action space in lateral navigation, and extend the approach to vertical control and two-aircraft avoidance. The key contributions include a practical mechanism to reduce instruction frequency, maintain safety, and scale RL to complex ATC tasks without changing the underlying MDP, enabling more realistic and tractable training. This technique offers a scalable pathway to operational ATC automation by bridging standard RL formulations with the realities of controller workflows and safety requirements.

Abstract

We introduce online action-stacking, an inference-time wrapper for reinforcement learning policies that produces realistic air traffic control commands while allowing training on a much smaller discrete action space. Policies are trained with simple incremental heading or level adjustments, together with an action-damping penalty that reduces instruction frequency and leads agents to issue commands in short bursts. At inference, online action-stacking compiles these bursts of primitive actions into domain-appropriate compound clearances. Using Proximal Policy Optimisation and the BluebirdDT digital twin platform, we train agents to navigate aircraft along lateral routes, manage climb and descent to target flight levels, and perform two-aircraft collision avoidance under a minimum separation constraint. In our lateral navigation experiments, action stacking greatly reduces the number of issued instructions relative to a damped baseline and achieves comparable performance to a policy trained with a 37-dimensional action space, despite operating with only five actions. These results indicate that online action-stacking helps bridge a key gap between standard reinforcement learning formulations and operational ATC requirements, and provides a simple mechanism for scaling to more complex control scenarios.
Paper Structure (23 sections, 9 equations, 19 figures, 6 tables)

This paper contains 23 sections, 9 equations, 19 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning and PPO
  4. Prior Work: RL Formulations of Air Traffic Control
  5. Motivation and Contribution
  6. Methodology
  7. Scenario Specification
  8. ATC Problem Statement
  9. Airspace
  10. Aircraft
  11. Problem Formulation
  12. State
  13. Action
  14. Reward
  15. Agent Training Configuration
  16. ...and 8 more sections

Figures (19)

  • Figure 1: The interaction loop of agent with environment in a Markov Decision Process.
  • Figure 2: The "X-Plus Sector", an artificial representation of en-route airspace with 20 nm airways and named navigation fixes.
  • Figure 3: Viable routes through the X-Plus sector from each starting point.
  • Figure 4: Policy controlling 2 aircraft through the X-Plus sector with no action damping penalty and no action stacking. A total of 92 separate actions were issued.
  • Figure 5: Number of actions taken per episode for a sample of $100$ episodes using an undamped lateral navigation policy for 2 aircraft navigation. A mean of 113.0 actions with standard deviation of 15.9 is observed.
  • ...and 14 more figures