Detecting and Mitigating System-Level Anomalies of Vision-Based Controllers

TL;DR

This work targets the safety of vision-based autonomous controllers by introducing a runtime anomaly monitor that leverages offline Hamilton-Jacobi reachability to identify system-level failure modes via the Backward Reachable Tube (BRT). A binary anomaly detector is trained on images labeled by offline reachability, enabling online detection of inputs likely to induce unsafe behavior, while a fallback controller preserves safety when anomalies are detected. The approach is validated on an autonomous aircraft taxiing task, demonstrating robust detection across diverse environments and outperforming component-level baselines such as prediction-error and ensemble methods. The results suggest a practical pathway to enhance robustness of vision-based control systems in the real world, especially where system-level safety must be guaranteed under unknown conditions.

Abstract

Autonomous systems, such as self-driving cars and drones, have made significant strides in recent years by leveraging visual inputs and machine learning for decision-making and control. Despite their impressive performance, these vision-based controllers can make erroneous predictions when faced with novel or out-of-distribution inputs. Such errors can cascade to catastrophic system failures and compromise system safety. In this work, we introduce a run-time anomaly monitor to detect and mitigate such closed-loop, system-level failures. Specifically, we leverage a reachability-based framework to stress-test the vision-based controller offline and mine its system-level failures. This data is then used to train a classifier that is leveraged online to flag inputs that might cause system breakdowns. The anomaly detector highlights issues that transcend individual modules and pertain to the safety of the overall system. We also design a fallback controller that robustly handles these detected anomalies to preserve system safety. We validate the proposed approach on an autonomous aircraft taxiing system that uses a vision-based controller for taxiing. Our results show the efficacy of the proposed approach in identifying and handling system-level anomalies, outperforming methods such as prediction error-based detection, and ensembling, thereby enhancing the overall safety and robustness of autonomous systems.

Figures (9)

• Figure 1: Detecting and mitigating system-level anomalies of vision based controller. Offline (top row): We compute the Backward Reachable Tube (BRT) of the vision-based system using Hamilton-Jacobi Reachability analysis chakraborty2023discovering. This allows us to create a labeled dataset of safe and unsafe images without any manual intervention. We train the anomaly detector using this dataset. Online (bottom row): We run the system on previously unseen environments where the trained anomaly detector triggers a fallback controller or the default controller depending on whether the observed image is anomalous.
• Figure 2: (a)$p_{x}$, $p_{y}$, $\theta$ denote the state of the aircraft; dashed-white lines show FoV of the camera. Runway simulation images with clear sky at (b) 9AM, (c) 5PM, and (d) 9PM, and overcast clouds at (e) 9AM, (f) 5PM, and (g) 9PM taken by camera mounted on the aircraft for the KMWH runway showing the variations in lighting conditions and shadows ((c),(f)) for changes in the environment.
• Figure 3: BRT of 9AM KMWH runway (part of train set) shown on the left along with a few training images and BRT of 5PM KSFO runway (part of test set) shown on the right, along with a few testing images, showing diversity in our training and testing scenarios.
• Figure 4: Failures detected by AD. (a, b) Images correspond to the aircraft being close to the runway boundaries (highlighted with the magenta bounding boxes).(c, d) The visual controller confuses the runway markings (highlighted with the cyan bounding boxes) with the centerline and ultimately leads to a system failure. (e, f) Image (f) is (accurately) not classified as an anomaly during the night time (the same image is classified as anomaly during the day, shown in (e)), as the runway lights (highlighted with the yellow bounding boxes) help the visual controller to predict its position accurately and thereby avoid failure.
• Figure 5: (a) Comparison between prediction error (blue) and BRT-based (red) labels. (b) Prediction error-based labels for $threshold=0.3$ (green) and $threshold=0.6$ (red). (c) Yellow and Green lines show trajectories starting from the yellow and green stars, respectively.