Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-in-the-Loop Testing of AI Agents for Air Traffic Control with a Regulated Assessment Framework

Ben Carvell, Marc Thomas, Andrew Pace, Christopher Dorney, George De Ath, Richard Everson, Nick Pepper, Adam Keane, Samuel Tomlinson, Richard Cannon

TL;DR

A rigorous, human-in-the-loop evaluation framework for assessing the performance of AI agents on the task of Air Traffic Control, grounded in a regulator-certified simulator-based curriculum used for training and testing real-world trainee controllers.

Abstract

We present a rigorous, human-in-the-loop evaluation framework for assessing the performance of AI agents on the task of Air Traffic Control, grounded in a regulator-certified simulator-based curriculum used for training and testing real-world trainee controllers. By leveraging legally regulated assessments and involving expert human instructors in the evaluation process, our framework enables a more authentic and domain-accurate measurement of AI performance. This work addresses a critical gap in the existing literature: the frequent misalignment between academic representations of Air Traffic Control and the complexities of the actual operational environment. It also lays the foundations for effective future human-machine teaming paradigms by aligning machine performance with human assessment targets.

Human-in-the-Loop Testing of AI Agents for Air Traffic Control with a Regulated Assessment Framework

TL;DR

A rigorous, human-in-the-loop evaluation framework for assessing the performance of AI agents on the task of Air Traffic Control, grounded in a regulator-certified simulator-based curriculum used for training and testing real-world trainee controllers.

Abstract

We present a rigorous, human-in-the-loop evaluation framework for assessing the performance of AI agents on the task of Air Traffic Control, grounded in a regulator-certified simulator-based curriculum used for training and testing real-world trainee controllers. By leveraging legally regulated assessments and involving expert human instructors in the evaluation process, our framework enables a more authentic and domain-accurate measurement of AI performance. This work addresses a critical gap in the existing literature: the frequent misalignment between academic representations of Air Traffic Control and the complexities of the actual operational environment. It also lays the foundations for effective future human-machine teaming paradigms by aligning machine performance with human assessment targets.
Paper Structure (21 sections, 12 figures, 4 tables)

This paper contains 21 sections, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Prior Work: ATC Automation Research
  3. Prior Work: Human-in-the-Loop and Curriculum-Driven Evaluation
  4. Motivation and Contribution
  5. Methodology: Machine Basic Training (MBT)
  6. The 'Area Basic' Training Course
  7. Legal Foundations of Training
  8. Scope for Agent Development and Assessment
  9. Formative and Summative Exercises
  10. Assessment Procedure
  11. Candidate Agents
  12. Rules-based Agent Hawk
  13. Optimisation-based Agent Falcon
  14. Experimental Validation
  15. Inter-Rater Reliability (IRR) Assessment
  16. ...and 6 more sections

Figures (12)

  • Figure 1: The progression of training to become a licensed ATCO at NATS NATS_ATCO_pathway.
  • Figure 2: Judgement of safety in plan view. In scenario A (left) BAW123 is in conflict with AEU666, with no lateral or vertical separation ensured, meaning that a risk of future collision exists. In scenario B (middle) BAW123 has been climbed to a level 10 flight levels (1000 feet) below AEU666, ensuring safety between them.
  • Figure 3: Judgement of effective controlling in plan view. In scenario A (left) BAW123 is unable to climb due to a potential conflict with AEU666. In scenario B (middle) AEU666 has been turned behind such that BAW123 can climb without risk of collision.
  • Figure 4: The flow down of requirements from top-level CAA regulations to machine basic.
  • Figure 5: Rules-based agent Hawk controlling an exercise in the BluebirdDT simulation environment.
  • ...and 7 more figures