Out-of-Distribution Detection for Safety Assurance of AI and Autonomous Systems

TL;DR

This paper addresses the challenge of ensuring safety for AI-enabled autonomous systems operating in open-world settings by focusing on Out-of-Distribution (OOD) detection as a core safety mechanism. It offers a generalized OOD framework that encompasses open-set, novelty, and anomaly detection and classifies techniques across data management, model learning, verification, and runtime monitoring, with a taxonomy based on filters vs wrappers and on unsupervised, supervised, semi-supervised, and RL paradigms. By mapping OOD methods to the autonomous system lifecycle, the work provides a structured view of how to argue safety through evidence, safety cases, and through-life monitoring, while highlighting data quality, benchmarks, metrics, explainability, and the emerging role of Foundation Models. Key contributions include a comprehensive taxonomy, lifecycle integration guidance, and a discussion of practical challenges and future directions needed to achieve robust, verifiable safety in open-world autonomous systems. The findings emphasize that safety guarantees depend on multi-faceted, system-centric OOD monitoring, continuous data and requirement updates, and careful handling of semantic gaps, computational constraints, and explainability across all lifecycle stages.

Abstract

The operational capabilities and application domains of AI-enabled autonomous systems have expanded significantly in recent years due to advances in robotics and machine learning (ML). Demonstrating the safety of autonomous systems rigorously is critical for their responsible adoption but it is challenging as it requires robust methodologies that can handle novel and uncertain situations throughout the system lifecycle, including detecting out-of-distribution (OoD) data. Thus, OOD detection is receiving increased attention from the research, development and safety engineering communities. This comprehensive review analyses OOD detection techniques within the context of safety assurance for autonomous systems, in particular in safety-critical domains. We begin by defining the relevant concepts, investigating what causes OOD and exploring the factors which make the safety assurance of autonomous systems and OOD detection challenging. Our review identifies a range of techniques which can be used throughout the ML development lifecycle and we suggest areas within the lifecycle in which they may be used to support safety assurance arguments. We discuss a number of caveats that system and safety engineers must be aware of when integrating OOD detection into system lifecycles. We conclude by outlining the challenges and future work necessary for the safe development and operation of autonomous systems across a range of domains and applications.

Figures (3)

• Figure 1: The ML lifecycle showing the integration of ML development activities with safety assurance activities, platform engineering activities and run-time operation. Adapted from ashmore2021assuringamlas2021
• Figure 2: The autonomous system's operational environment is the ODD (C). Region $M$ is ID for the ML model. However, there are regions of uncertainty: the grey area of C outside $M$ and the regions labelled B within $M$burton2023addressing caused by insufficient training data or erroneous data for region C, and, insufficient learning or erroneous learning for B. Regions B and the grey area of C are thus OOD w.r.t. $M$ but inside the ODD. The pink area outside C is OOD w.r.t. the ODD. For level 5 autonomy the ODD covers the real world so C=W.
• Figure 3: Effects of distributional shift on a supervised classification problem black circle, red crosses and orange boxes are classes defined on two features ($f_1$ and $f_2$). The dashed line represents ground truth class separation boundaries.