Gaussian-Based and Outside-the-Box Runtime Monitoring Join Forces

TL;DR

Neural networks can produce incorrect predictions even when confident, especially on out-of-model-scope data, which motivates runtime OMS monitoring in safety-critical settings. The authors propose a hybrid monitoring approach that combines a Gaussian Monitor with an Outside-the-Box, cluster-based monitor, augmented by gradient-based neuron selection to reduce monitoring load. Extending gaussianMonitor with clustering information and evaluating on CIFAR-10 and GTSRB, they show the hybrid monitor improves OMS detection on more complex data and maintains efficiency by reducing the number of monitored neurons. Across datasets, clustering activation values before applying the Gaussian monitor yields better performance than the unclustered baseline, and selective neuron monitoring provides substantial computation savings with modest performance loss. Together, the work advances practical, efficient OMS monitoring for real-time safety-critical deployments.

Abstract

Since neural networks can make wrong predictions even with high confidence, monitoring their behavior at runtime is important, especially in safety-critical domains like autonomous driving. In this paper, we combine ideas from previous monitoring approaches based on observing the activation values of hidden neurons. In particular, we combine the Gaussian-based approach, which observes whether the current value of each monitored neuron is similar to typical values observed during training, and the Outside-the-Box monitor, which creates clusters of the acceptable activation values, and, thus, considers the correlations of the neurons' values. Our experiments evaluate the achieved improvement.

Figures (6)

• Figure 1: Comparison of different monitors. The axes represent the neurons and their activation values, and the points are the vectors of activation values observed for a particular input. Boxes depict safe areas of the Box and the combined monitor. The points below each axis show the activation values mapped to the specific axis with $x$ highlighting the mean. The boxes below the axes depict the activation values the Gaussian monitor accepts.
• Figure 2: Illustration of the CIFAR-10 cifar10 dataset.
• Figure 3: Illustration of the GTSRB gtsrb dataset.
• Figure 4: Illustration of various manipulations performed on the original id data of the CIFAR-10 dataset to create the oms datasets.
• Figure 5: nn architecture for classifying images of the CIFAR-10 dataset developed by Vishal-Ramesh-NN-Architecture-Cifar10.