Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Run-time Safety Monitors for Machine Learning Components

Ozan Vardal, Richard Hawkins, Colin Paterson, Chiara Picardi, Daniel Omeiza, Lars Kunze, Ibrahim Habli

TL;DR

The paper tackles runtime safety assurance for ML components in autonomous systems when ground truth is unavailable. It introduces a seven-step methodology to build ML safety monitors from degraded data by mapping environmental degradations to safety levels and training a monitor to predict safety risk in real time. The authors validate the approach on a road-sign classification task (GTSRB) using haze and blur perturbations, achieving a 92% monitor accuracy under 5-fold cross-validation and demonstrating actionable safety responses. Overall, the work provides a general, transfer-assurance–driven framework for maintaining ML safety under post-deployment changes, with potential applicability beyond image data to other domains.

Abstract

For machine learning components used as part of autonomous systems (AS) in carrying out critical tasks it is crucial that assurance of the models can be maintained in the face of post-deployment changes (such as changes in the operating environment of the system). A critical part of this is to be able to monitor when the performance of the model at runtime (as a result of changes) poses a safety risk to the system. This is a particularly difficult challenge when ground truth is unavailable at runtime. In this paper we introduce a process for creating safety monitors for ML components through the use of degraded datasets and machine learning. The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output. We demonstrate the viability of our approach through some initial experiments using publicly available speed sign datasets.

Learning Run-time Safety Monitors for Machine Learning Components

TL;DR

The paper tackles runtime safety assurance for ML components in autonomous systems when ground truth is unavailable. It introduces a seven-step methodology to build ML safety monitors from degraded data by mapping environmental degradations to safety levels and training a monitor to predict safety risk in real time. The authors validate the approach on a road-sign classification task (GTSRB) using haze and blur perturbations, achieving a 92% monitor accuracy under 5-fold cross-validation and demonstrating actionable safety responses. Overall, the work provides a general, transfer-assurance–driven framework for maintaining ML safety under post-deployment changes, with potential applicability beyond image data to other domains.

Abstract

For machine learning components used as part of autonomous systems (AS) in carrying out critical tasks it is crucial that assurance of the models can be maintained in the face of post-deployment changes (such as changes in the operating environment of the system). A critical part of this is to be able to monitor when the performance of the model at runtime (as a result of changes) poses a safety risk to the system. This is a particularly difficult challenge when ground truth is unavailable at runtime. In this paper we introduce a process for creating safety monitors for ML components through the use of degraded datasets and machine learning. The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output. We demonstrate the viability of our approach through some initial experiments using publicly available speed sign datasets.
Paper Structure (7 sections, 4 equations, 5 figures, 1 table)

This paper contains 7 sections, 4 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Evaluation
  5. Discussion
  6. Conclusions
  7. Acknowledgements

Figures (5)

  • Figure 1: Three Stage Transfer Assurance Process with activities shown in green and artefacts in yellow.
  • Figure 2: Overview of process for creation of ML safety monitors
  • Figure 3: Perturbed samples from the GTSRB dataset showing the independent effects of the influencing factors. Numbers under the images indicate the level of the effect applied.
  • Figure 4: Heatmap of accuracy of the original road sign classifier for each degraded dataset. Axes represent the $\epsilon$ (degradation level) of each degradation effect. The region with the green border represents epsilon combinations that result in $accuracy \geq70\%$, while those with the amber and red border represent epsilon combinations that result in $40\%\leq accuracy<70\%$ and $accuracy<40\%$ respectively.
  • Figure 5: The left panel shows the performance of the safety monitor across three classes, with each class representing the anticipated operational safety of the ML component in environments with different levels of perturbation. The right panel shows Receiver Operating Characteristic (ROC) curves for each of the three operational safety labels classified by the safety monitor.