Peaking into the Black-box: Prediction Intervals Give Insight into Data-driven Quadrotor Model Reliability

TL;DR

The paper tackles the challenge of quantifying reliability for data-driven quadrotor aerodynamic models, including grey- and black-box types, by using prediction intervals (PIs) to bound outputs under uncertainty. It compares three PI estimation strategies—polynomial analytic PIs, ANN bootstrap, and ANN quality-driven PIs—via a high-fidelity simulator and then applies them to real high-speed flight data to study how PIs respond to interpolation versus extrapolation. Key findings show that all methods produce valid $95\%$ PIs under Gaussian noise, with ANN-based PIs widening in extrapolation (and often narrowing with interpolation), while polynomial PIs are less sensitive to interpolation but can be conservative in extrapolation; bootstrap typically offers practical balance. The work provides probabilistic bounds that help assess model reliability, inform safe control, and guide future development of data-driven quadrotor models and their certification in aerospace contexts.

Abstract

Ensuring the reliability and validity of data-driven quadrotor model predictions is essential for their accepted and practical use. This is especially true for grey- and black-box models wherein the mapping of inputs to predictions is not transparent and subsequent reliability notoriously difficult to ascertain. Nonetheless, such techniques are frequently and successfully used to identify quadrotor models. Prediction intervals (PIs) may be employed to provide insight into the consistency and accuracy of model predictions. This paper estimates such PIs for polynomial and Artificial Neural Network (ANN) quadrotor aerodynamic models. Two existing ANN PI estimation techniques - the bootstrap method and the quality driven method - are validated numerically for quadrotor aerodynamic models using an existing high-fidelity quadrotor simulation. Quadrotor aerodynamic models are then identified on real quadrotor flight data to demonstrate their utility and explore their sensitivity to model interpolation and extrapolation. It is found that the ANN-based PIs widen considerably when extrapolating and remain constant, or shrink, when interpolating. While this behaviour also occurs for the polynomial PIs, it is of lower magnitude. The estimated PIs establish probabilistic bounds within which the quadrotor model outputs will likely lie, subject to modelling and measurement uncertainties that are reflected through the PI widths.

Figures (16)

• Figure 1: Quadrotor body reference frame.
• Figure 2: Quadrotor test platform, the HDBeetle.
• Figure 3: Example trajectories (orange) of flights employed for model identification. Also shown are the planar trajectories (yellow) of these manoeuvres, along with some snapshots of the quadrotor's orientation at various points in the trajectory.
• Figure 4: Illustration of the 95% prediction interval (PI) performances of each of the PI estimation techniques - Polynomial (orange), ANN bootstrap (Cyan), ANN quality driven (Blue) - with respect to Gaussian uncertainty in the $F_{z}$ measurements (grey) and model inputs, which culminate in model prediction variation (pink).
• Figure 5: Illustration of the 95% prediction interval (PI) performances of each of the PI estimation techniques - Polynomial (orange), ANN bootstrap (Cyan), ANN quality driven (Blue) - with respect to Gaussian uncertainty in the $F_{x}$ measurements (grey) and model inputs, which culminate in model prediction variation (pink).