Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Learning Safety Concerns in Automated Driving Perception

Stephanie Abrecht, Alexander Hirsch, Shervin Raafatnia, Matthias Woehrle

TL;DR

This paper refines the safety concerns framework for DNN-based perception in automated driving by expanding prior concerns to fourteen and organizing them into four source-based categories aligned with SOTIF. It emphasizes the open-world nature of driving, data and dataset preparation, DNN-specific properties, and evaluation challenges, arguing that safety arguments require safety-aware metrics and long-tail risk assessment. The work provides definitions, mappings to ISO/SAE safety concepts, and guidance for cross-functional teams to identify, communicate, and mitigate risks in perception pipelines. The practical impact is a structured problem space that informs mitigations, evaluation strategies, and collaboration across safety, engineering, and ML teams to reduce residual risk in real-world AD deployments.

Abstract

Recent advances in the field of deep learning and impressive performance of deep neural networks (DNNs) for perception have resulted in an increased demand for their use in automated driving (AD) systems. The safety of such systems is of utmost importance and thus requires to consider the unique properties of DNNs. In order to achieve safety of AD systems with DNN-based perception components in a systematic and comprehensive approach, so-called safety concerns have been introduced as a suitable structuring element. On the one hand, the concept of safety concerns is -- by design -- well aligned to existing standards relevant for safety of AD systems such as ISO 21448 (SOTIF). On the other hand, it has already inspired several academic publications and upcoming standards on AI safety such as ISO PAS 8800. While the concept of safety concerns has been previously introduced, this paper extends and refines it, leveraging feedback from various domain and safety experts in the field. In particular, this paper introduces an additional categorization for a better understanding as well as enabling cross-functional teams to jointly address the concerns.

Deep Learning Safety Concerns in Automated Driving Perception

TL;DR

This paper refines the safety concerns framework for DNN-based perception in automated driving by expanding prior concerns to fourteen and organizing them into four source-based categories aligned with SOTIF. It emphasizes the open-world nature of driving, data and dataset preparation, DNN-specific properties, and evaluation challenges, arguing that safety arguments require safety-aware metrics and long-tail risk assessment. The work provides definitions, mappings to ISO/SAE safety concepts, and guidance for cross-functional teams to identify, communicate, and mitigate risks in perception pipelines. The practical impact is a structured problem space that informs mitigations, evaluation strategies, and collaboration across safety, engineering, and ML teams to reduce residual risk in real-world AD deployments.

Abstract

Recent advances in the field of deep learning and impressive performance of deep neural networks (DNNs) for perception have resulted in an increased demand for their use in automated driving (AD) systems. The safety of such systems is of utmost importance and thus requires to consider the unique properties of DNNs. In order to achieve safety of AD systems with DNN-based perception components in a systematic and comprehensive approach, so-called safety concerns have been introduced as a suitable structuring element. On the one hand, the concept of safety concerns is -- by design -- well aligned to existing standards relevant for safety of AD systems such as ISO 21448 (SOTIF). On the other hand, it has already inspired several academic publications and upcoming standards on AI safety such as ISO PAS 8800. While the concept of safety concerns has been previously introduced, this paper extends and refines it, leveraging feedback from various domain and safety experts in the field. In particular, this paper introduces an additional categorization for a better understanding as well as enabling cross-functional teams to jointly address the concerns.
Paper Structure (27 sections, 2 figures, 1 table)

This paper contains 27 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Operational Design Domain
  4. Risk in Safety and Machine Learning
  5. Safety Concerns Categorization
  6. Definition of Safety Concerns
  7. Categorization of Safety Concerns
  8. Open-World Context Concerns
  9. Highly Limited Specifiability of Operational Design Domain
  10. Insufficient Coverage of the Open-World ODD
  11. Distributional Shift Over Time
  12. Data and Data Set Preparation
  13. Leakage in the Development Cycle
  14. Insufficient Labeling Quality
  15. Susceptibility to (Data) Biases
  16. ...and 12 more sections

Figures (2)

  • Figure 1: The terminology used in this paper, which is aligned with ISO 21448 (SOTIF) sotif: A safety concern can lead to a functional insufficiency within a DNN. Once the functional insufficiency is triggered by a triggering condition, it results in an output insufficiency of the DNN. Output insufficiencies may lead to hazardous behavior of the system.
  • Figure 2: Safety concerns categorization overview.