AdaptiveCoPilot: Design and Testing of a NeuroAdaptive LLM Cockpit Guidance System in both Novice and Expert Pilots

TL;DR

This work tackles cognitive load in modern cockpit operations by introducing AdaptiveCoPilot, a neuroadaptive guidance system that real-time adjusts multimodal feedback based on cognitive states measured with fNIRS. It leverages a context-aware PHI-3 large language model to reason over cognitive-state input and task context, guiding modality and information-load choices during preflight tasks in a VR UH-60 cockpit. A formative study with expert pilots informs design requirements, and an eight-pilot VR evaluation provides initial evidence that adaptive guidance increases optimal memory states and reduces completion time, with mixed effects on perception and attention. The findings suggest neuroadaptive guidance can enhance training and safety in aviation, while highlighting challenges in state differentiation, error recovery, and user trust that warrant further refinement and larger-scale testing.

Abstract

Pilots operating modern cockpits often face high cognitive demands due to complex interfaces and multitasking requirements, which can lead to overload and decreased performance. This study introduces AdaptiveCoPilot, a neuroadaptive guidance system that adapts visual, auditory, and textual cues in real time based on the pilot's cognitive workload, measured via functional Near-Infrared Spectroscopy (fNIRS). A formative study with expert pilots (N=3) identified adaptive rules for modality switching and information load adjustments during preflight tasks. These insights informed the design of AdaptiveCoPilot, which integrates cognitive state assessments, behavioral data, and adaptive strategies within a context-aware Large Language Model (LLM). The system was evaluated in a virtual reality (VR) simulated cockpit with licensed pilots (N=8), comparing its performance against baseline and random feedback conditions. The results indicate that the pilots using AdaptiveCoPilot exhibited higher rates of optimal cognitive load states on the facets of working memory and perception, along with reduced task completion times. Based on the formative study, experimental findings, qualitative interviews, we propose a set of strategies for future development of neuroadaptive pilot guidance systems and highlight the potential of neuroadaptive systems to enhance pilot performance and safety in aviation environments.

Figures (6)

• Figure 1: The front and backend systems. A change in arrow color represents a change in information.
• Figure 2: Cognitive classifier: fNIRS data is preprocessed using wavelet filters over a sliding window then the 3 cognitive facets are classified.
• Figure 3: All Workload Across Conditions
• Figure 4: Optimal Workload Across Conditions and Procedures. We see a optimal increase in the adaptive condition for memory. A-I corresponds to a procedure.
• Figure 5: Error Rates Across Conditions and Procedures. We see a slight decrease in error rates in the baseline. A-I corresponds to a procedure.