A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning

TL;DR

A novel Mahalanobis distance-based (MD) anomaly detection framework, called MDX, for deep RL algorithms that simultaneously addresses random, adversarial, and out-of-distribution state outliers in both offline and online settings and is extended to robust and distribution-free versions by incorporating Robust MD and conformal inference techniques.

Abstract

In deep reinforcement learning (RL) systems, abnormal states pose significant risks by potentially triggering unpredictable behaviors and unsafe actions, thus impeding the deployment of RL systems in real-world scenarios. It is crucial for reliable decision-making systems to have the capability to cast an alert whenever they encounter unfamiliar observations that they are not equipped to handle. In this paper, we propose a novel Mahalanobis distance-based (MD) anomaly detection framework, called \textit{MDX}, for deep RL algorithms. MDX simultaneously addresses random, adversarial, and out-of-distribution (OOD) state outliers in both offline and online settings. It utilizes Mahalanobis distance within class-conditional distributions for each action and operates within a statistical hypothesis testing framework under the Gaussian assumption. We further extend it to robust and distribution-free versions by incorporating Robust MD and conformal inference techniques. Through extensive experiments on classical control environments, Atari games, and autonomous driving scenarios, we demonstrate the effectiveness of our MD-based detection framework. MDX offers a simple, unified, and practical anomaly detection tool for enhancing the safety and reliability of RL systems in real-world applications.

Figures (23)

• Figure 1: (a) An autonomous car navigates using location data observed from sensors such as GPS. Without an effective anomaly detection mechanism, inaccuracies or malfunctions in these sensors can cause the car to prematurely turn right, leading to a collision. (b) and (c): Performance degradation occurs when noisy states are observed in the Breakout environment. Gaussian noises with increasing standard deviations are injected into the state observations during policy deployment (b) and policy learning (c).
• Figure 2: The detection pipeline of MDX. We feed the state into the policy network to extract the feature vector and identify its class. For each class, we estimate $(\mu,\Sigma)$ and establish a detection threshold depicted as a dashed ellipse. To determine whether a new state is an outlier, we evaluate its features and compute the distance to the class centroids. If the distance falls below the set threshold, the state is classified as an inlier (green points). Conversely, the state is marked as an outlier (red points).
• Figure 3: Contours under the estimation based on MD and Robust MD across different outlier types on Breakout. Black and red points denote inliers and outliers, respectively. The dimension of state feature vectors after a pre-trained PPO policy is reduced by t-SNE van2008visualizing.
• Figure 4: Performance on MountainCar across various state outliers in online learning. The first row shows the policy performance during learning. The second row shows the relationship between the averaged detection accuracy and performance.
• Figure 5: Performance on Tutankham across various state outliers in online learning. The first row shows the policy performance during learning. The second row shows the relationship between the averaged detection accuracy and achieved final performance.

Theorems & Definitions (2)

• Proposition 1
• proof