Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safety Monitoring of Machine Learning Perception Functions: a Survey

Raul Sena Ferreira, Joris Guérin, Kevin Delmas, Jérémie Guiochet, Hélène Waeselynck

TL;DR

This survey tackles the safety monitoring of ML-based perception in safety-critical systems by adopting a top-down, hazard-analysis–driven framework. It develops a comprehensive taxonomy of runtime threats (in-distribution errors, novelty, distributional shift, and adversarial inputs) and maps these to both internal and external detection mechanisms, including uncertainty estimation, domain knowledge, input/output monitoring, and multi-sensor coherence. It further surveys recovery strategies and evaluation practices, emphasizing system-level performance, certification needs, and standardized benchmarks. The work provides a structured guide for safety practitioners and ML researchers to design robust SMs, integrate them into real-time systems, and pursue certification under evolving safety standards.

Abstract

Machine Learning (ML) models, such as deep neural networks, are widely applied in autonomous systems to perform complex perception tasks. New dependability challenges arise when ML predictions are used in safety-critical applications, like autonomous cars and surgical robots. Thus, the use of fault tolerance mechanisms, such as safety monitors, is essential to ensure the safe behavior of the system despite the occurrence of faults. This paper presents an extensive literature review on safety monitoring of perception functions using ML in a safety-critical context. In this review, we structure the existing literature to highlight key factors to consider when designing such monitors: threat identification, requirements elicitation, detection of failure, reaction, and evaluation. We also highlight the ongoing challenges associated with safety monitoring and suggest directions for future research.

Safety Monitoring of Machine Learning Perception Functions: a Survey

TL;DR

This survey tackles the safety monitoring of ML-based perception in safety-critical systems by adopting a top-down, hazard-analysis–driven framework. It develops a comprehensive taxonomy of runtime threats (in-distribution errors, novelty, distributional shift, and adversarial inputs) and maps these to both internal and external detection mechanisms, including uncertainty estimation, domain knowledge, input/output monitoring, and multi-sensor coherence. It further surveys recovery strategies and evaluation practices, emphasizing system-level performance, certification needs, and standardized benchmarks. The work provides a structured guide for safety practitioners and ML researchers to design robust SMs, integrate them into real-time systems, and pursue certification under evolving safety standards.

Abstract

Machine Learning (ML) models, such as deep neural networks, are widely applied in autonomous systems to perform complex perception tasks. New dependability challenges arise when ML predictions are used in safety-critical applications, like autonomous cars and surgical robots. Thus, the use of fault tolerance mechanisms, such as safety monitors, is essential to ensure the safe behavior of the system despite the occurrence of faults. This paper presents an extensive literature review on safety monitoring of perception functions using ML in a safety-critical context. In this review, we structure the existing literature to highlight key factors to consider when designing such monitors: threat identification, requirements elicitation, detection of failure, reaction, and evaluation. We also highlight the ongoing challenges associated with safety monitoring and suggest directions for future research.

Paper Structure

This paper contains 52 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. What threats are being addressed by safety monitors?
  4. In-distribution errors
  5. Novelty threats
  6. Distributional shift threats
  7. Covariate shift
  8. Semantic shift
  9. Adversarial threats
  10. How to derive safety monitors from safety objectives?
  11. Which detection mechanisms can be used for safety monitoring?
  12. Internal mechanisms
  13. Uncertainty estimation
  14. Incorporating domain knowledge
  15. Learning with rejection
  16. ...and 37 more sections

Figures (3)

  • Figure 1: Safety Monitors for Machine Learning-based Perception Functions. In modern autonomous systems, state estimation provided by deep learning models cannot be trusted to make safety-critical decisions. Therefore, specific fault tolerance approaches should be implemented to ensure that failures of the ML perception function will not lead to catastrophic outcomes.
  • Figure 2: Key questions to design, implement, and deploy reliable safety monitors for ML perception. Each section of this work discusses a specific aspect of safety monitoring of machine learning perception.
  • Figure 3: Taxonomy of detection mechanisms. A visual representation of the different types of approaches to detect a failure of a critical ML-based perception function.