Safety Monitoring for Learning-Enabled Cyber-Physical Systems in Out-of-Distribution Scenarios

TL;DR

This work tackles safety assurance for learning-enabled cyber-physical systems under out-of-distribution inputs by directly monitoring safety through STL robustness predictions rather than detecting OOD inputs. It introduces a safety monitor that combines adaptive conformal prediction (ACP) with incremental learning (IL) to provide probabilistic guarantees while adapting to changing distributions, and it predicts future STL robustness values to flag potential safety violations. The approach is validated on two driving benchmarks (F1Tenth car with static obstacles and a race car with dynamic obstacles), demonstrating that ACP guarantees hold under distribution shift while IL improves recall, timeliness, and restores precision compared to baseline conformal methods. The results indicate that ACP+IL offers robust, timely safety monitoring for LE-CPS in real-world, non-stationary environments, with practical implications for deployment in autonomous systems and other safety-critical LE-CPS domains.

Abstract

The safety of learning-enabled cyber-physical systems is compromised by the well-known vulnerabilities of deep neural networks to out-of-distribution (OOD) inputs. Existing literature has sought to monitor the safety of such systems by detecting OOD data. However, such approaches have limited utility, as the presence of an OOD input does not necessarily imply the violation of a desired safety property. We instead propose to directly monitor safety in a manner that is itself robust to OOD data. To this end, we predict violations of signal temporal logic safety specifications based on predicted future trajectories. Our safety monitor additionally uses a novel combination of adaptive conformal prediction and incremental learning. The former obtains probabilistic prediction guarantees even on OOD data, and the latter prevents overly conservative predictions. We evaluate the efficacy of the proposed approach in two case studies on safety monitoring: 1) predicting collisions of an F1Tenth car with static obstacles, and 2) predicting collisions of a race car with multiple dynamic obstacles. We find that adaptive conformal prediction obtains theoretical guarantees where other uncertainty quantification methods fail to do so. Additionally, combining adaptive conformal prediction and incremental learning for safety monitoring achieves high recall and timeliness while reducing loss in precision. We achieve these results even in OOD settings and outperform alternative methods.

Figures (7)

• Figure 1: OOD inputs to the learning-enabled CPS component do not necessarily lead to safety violations. a) In the cartpole benchmark, a pole is attached to moving cart by a pivot point. The pole must be kept upright by applying left and right forces to the cart, while keeping the cart centered. b) We induce distribution shift in the cartpole's state trajectories by varying the environment parameters. The star indicates the in-distribution parameter selection. In this study, the reward measurement is a ground-truth evaluation of safety. The robustness value is a better indicator of the reward than the trajectory likelihood.
• Figure 2: Safety monitoring for learning-enabled cyber-physical systems. Observing only the black-boxed system's states, we employ a trajectory predictor, updated via incremental learning, to predict the system's future states. On this prediction, we use the STL and ACP frameworks to obtain a prediction region on the robustness value. A simple condition indicates whether a violation has been predicted.
• Figure 3: Case studies for empirical evaluation. a) A deep RL agent drives an F1Tenth car through series of hallways. Based on LIDAR measurements, the agent must select actions that avoid collisions with the walls. Our safety monitor predicts collisions with the walls. b) A deep RL agent traverses a race track. Based on grids that provide the locations of nearby vehicles and the road surface (the former is shown in this figure), the agent must select actions that avoid collisions with surrounding vehicles and keep the race car on the track. Our safety monitor predicts collisions with other vehicles.
• Figure 4: Empirical evaluations of Lemma \ref{['lem:lars_adaptive']} and Theorem \ref{['thm:satisfy_safety']} for both case studies (without IL). For more accurate estimates, values are calculated over the union of all 10 trials. Lemma \ref{['lem:lars_adaptive']} (top): the empirical ACP coverage rates are within the theoretical bounds in the ID and OOD scenarios. Theorem \ref{['thm:satisfy_safety']} (bottom): for ID and OOD simulations where the assumptions of Theorem \ref{['thm:satisfy_safety']} hold, the empirical STL satisfaction rates are within the theoretical bounds. Sudden drops occur at times when the system reaches a region that is challenging to safely navigate (e.g., sharp corners).
• Figure 5: Recall, precision, and timeliness of our safety monitor for Case Studies I (top) and II (bottom), recorded over 10 trials. For Case Study I, the OOD scenarios are 3 missing LIDAR rays, 5 missing LIDAR rays, additive uniform (0,0.9) noise, and additive uniform (0,1.0) noise. For Case Study II, the OOD scenarios are 2, 3, 4, and 5 dynamic obstacles (obs.) on the race track.

Theorems & Definitions (5)

• Proposition 1: acp, acp
• Lemma 1
• proof
• Theorem 1
• proof