Conformal Safety Monitoring for Flight Testing: A Case Study in Data-Driven Safety Learning

TL;DR

This paper addresses runtime safety monitoring for flight testing under uncertain dynamics by proposing a three-stage data-driven framework: (1) forecast the future state from a short observation history using a linear predictor, (2) classify the predicted future state as safe or unsafe with a nearest-neighbor model, and (3) calibrate the classifier with conformal prediction to provide a statistically guaranteed miss rate $\epsilon$. Offline data collection of unsafe and safe trajectories enables learning a predictive representation and a calibrated safety threshold $s^*$, with a corresponding p-value $\epsilon^*$ used at runtime to prompt preemptive aborts when necessary. The approach demonstrates reliable identification of unsafe scenarios and adherence to theoretical guarantees on a flight dynamics model with uncertain parameters, outperforming several baselines. The method offers a practical, interpretable mechanism for data-driven safety learning in high-stakes human-in-the-loop settings, potentially extending to more complex safety criteria beyond lateral acceleration thresholds.

Abstract

We develop a data-driven approach for runtime safety monitoring in flight testing, where pilots perform maneuvers on aircraft with uncertain parameters. Because safety violations can arise unexpectedly as a result of these uncertainties, pilots need clear, preemptive criteria to abort the maneuver in advance of safety violation. To solve this problem, we use offline stochastic trajectory simulation to learn a calibrated statistical model of the short-term safety risk facing pilots. We use flight testing as a motivating example for data-driven learning/monitoring of safety due to its inherent safety risk, uncertainty, and human-interaction. However, our approach consists of three broadly-applicable components: a model to predict future state from recent observations, a nearest neighbor model to classify the safety of the predicted state, and classifier calibration via conformal prediction. We evaluate our method on a flight dynamics model with uncertain parameters, demonstrating its ability to reliably identify unsafe scenarios, match theoretical guarantees, and outperform baseline approaches in preemptive classification of risk.

Figures (2)

• Figure 1: True versus Predicted Outputs for Unsafe/Safe Trajectories with Associated $p$-Values
• Figure 2: Miss Rate and Classification Power for Varying $\epsilon$