Operational range bounding of spectroscopy models with anomaly detection

TL;DR

Isolation Forests are shown to effectively identify contexts where prediction models are likely to fail and the best performance is seen when Isolation Forests model projections of the prediction model's explainability SHAP values are seen.

Abstract

Safe operation of machine learning models requires architectures that explicitly delimit their operational ranges. We evaluate the ability of anomaly detection algorithms to provide indicators correlated with degraded model performance. By placing acceptance thresholds over such indicators, hard boundaries are formed that define the model's coverage. As a use case, we consider the extraction of exoplanetary spectra from transit light curves, specifically within the context of ESA's upcoming Ariel mission. Isolation Forests are shown to effectively identify contexts where prediction models are likely to fail. Coverage/error trade-offs are evaluated under conditions of data and concept drift. The best performance is seen when Isolation Forests model projections of the prediction model's explainability SHAP values.

Figures (3)

• Figure 1: Transit light curves for the same star/planet system, produced by the simulation pipelines of the ADC19 morello2020exotethyswaldmann2015tau (left) and ADC21 mugnai2020arielradsarkar2021exosim (middle) challenges (seen: 55 light curves, one per wavelength channel, transformed through a sliding-window aggregation over time and across the 10 photon noise instances of the star's first stellar spot configuration). Right plot: the planet's transmission spectrum (55 channels) the trained model needs to predict from the 55 light curves. Simulations for exoplanet Qatar-4b, using star/planet parameters from the Ariel Mission Candidate Sample MCS:edwards2019updatedMCS:edwards2022ariel.
• Figure 2: Relation between anomaly scores produced by Isolation Forests and predictive performance of another model trained over the same data. Results obtained through 10-fold cross-validation over the ADC21 dataset. Plots show values measured across the validation-folds for the whole dataset, from Isolation Forest, PCA and Ridge models trained over the respective train-folds. In the left plot, the Isolation Forest models the distribution of $\mathbf{X}$, the 611-dimensional representations of light curves that the Ridge model takes as inputs. In the central plot the modelled distribution is that of $\mathbf{Y}_{true}$, the spectra that Ridge models are trained to predict. In the right plot the Isolation Forest models the $\mathbf{X}_{shap}$ values produced by the SHAP Explainable AI library SHAP that represent how the Ridge model transforms $\mathbf{X}$ into $\mathbf{Y}_{pred}$.
• Figure 3: Trade-offs achievable between model coverage and predictive performance when acceptance thresholds are defined over Isolation Forests' anomaly scores. Averages across the trade-offs seen in individual validation folds.