Explaining Unreliable Perception in Automated Driving: A Fuzzy-based Monitoring Approach

TL;DR

This work addresses unreliable ML perception in autonomous driving caused by varying environmental conditions by introducing a fuzzy-based runtime monitor that produces human-interpretable PO-condition explanations and can enforce safe operation. The approach learns an interpretable classifier $oldsymbol{\Xi}$ via online evolving fuzzy systems (AlMMo), forming dataclouds with prototypes and rules that map PO conditions to misperception, and then uses these rules as a runtime monitor $m_C$ to detect untrustworthy conditions and gather evidence for an assurance case. Through a case study on the SCA scenario using the BDD100k dataset and yolov5x6, the authors demonstrate both qualitative interpretability (ISO34503-compliant ODD specification) and quantitative runtime-safety benefits, showing improved safety with low impact on availability compared to baselines. The results support the viability of interpretable, online-updatable monitors for online fault tolerance and safety assurance in evolving ML perception components, with future work on DevOps integration and broader ODD expansion.

Abstract

Autonomous systems that rely on Machine Learning (ML) utilize online fault tolerance mechanisms, such as runtime monitors, to detect ML prediction errors and maintain safety during operation. However, the lack of human-interpretable explanations for these errors can hinder the creation of strong assurances about the system's safety and reliability. This paper introduces a novel fuzzy-based monitor tailored for ML perception components. It provides human-interpretable explanations about how different operating conditions affect the reliability of perception components and also functions as a runtime safety monitor. We evaluated our proposed monitor using naturalistic driving datasets as part of an automated driving case study. The interpretability of the monitor was evaluated and we identified a set of operating conditions in which the perception component performs reliably. Additionally, we created an assurance case that links unit-level evidence of \textit{correct} ML operation to system-level \textit{safety}. The benchmarking demonstrated that our monitor achieved a better increase in safety (i.e., absence of hazardous situations) while maintaining availability (i.e., ability to perform the mission) compared to state-of-the-art runtime ML monitors in the evaluated dataset.

Figures (6)

• Figure 1: Overview of Proposed Fuzzy-based Monitor Learning
• Figure 2: Learning Monitor: Datacloud Discovery
• Figure 3: Representing PO conditions with dataclouds
• Figure 4: Representative images of the discovered dataclouds
• Figure 5: ISO34503 compliant ODD Specification