Detection of Unknown Errors in Human-Centered Systems

TL;DR

This paper addresses the challenge of detecting unknown errors in safety-critical human-centered AI systems that lack predefined error signatures. It introduces a two-stage, model-agnostic framework that learns physics-guided surrogate coefficients via a dynamics-induced hybrid RNN (DiH-RNN) and then applies conformal inference on the coefficient vector $\omega$ to establish a conformal range $d$, enabling STL-based safety checks on the coefficient model. By focusing conformance on model coefficients rather than outputs, the method aims to detect unknown-unknown errors earlier than traditional runtime monitors, with concrete penalties captured by the robustness of STL properties. Applied to automated insulin delivery, aircraft pitch control, and autonomous driving, the approach achieves early, high-precision detection with reported $PPV$ values up to $100\%$ across multiple unknown-error scenarios, demonstrating practical potential for safer real-world deployments. The work highlights the value of coefficient-centric, physics-guided monitoring for improving safety in real-time, model-agnostic settings and suggests directions for data-efficiency and broader validation.

Abstract

Artificial Intelligence-enabled systems are increasingly being deployed in real-world safety-critical settings involving human participants. It is vital to ensure the safety of such systems and stop the evolution of the system with error before causing harm to human participants. We propose a model-agnostic approach to detecting unknown errors in such human-centered systems without requiring any knowledge about the error signatures. Our approach employs dynamics-induced hybrid recurrent neural networks (DiH-RNN) for constructing physics-based models from operational data, coupled with conformal inference for assessing errors in the underlying model caused by violations of physical laws, thereby facilitating early detection of unknown errors before unsafe shifts in operational data distribution occur. We evaluate our framework on multiple real-world safety critical systems and show that our technique outperforms the existing state-of-the-art in detecting unknown errors.

Figures (4)

• Figure 1: System Model of Human-Centered Systems. In this architecture, the human operator can be both part of the control mechanism and within the operational dynamics of the plant itself. The plant's state is monitored through sensors and control actions are performed via actuators, processes that are prone to inaccuracies and errors.
• Figure 2: Overview of the Proposed Approach: The diagram illustrates the two-stage process of our methodology. The physics-guided surrogate models facilitate the determination of a conformal range for the surrogate model coefficients. Subsequently, in the Operational Phase, another physics-guided model is learned using real-time operational traces. To ensure the model's conformance, the critical assessment in this phase involves verifying whether the coefficients of this operational model are within the conformal range identified during the training phase.
• Figure 3: The figure illustrates a comparative analysis between current runtime monitors and our approach to error detection. While existing techniques can detect errors at 30 when the safety threshold is breached, our approach can identify errors at 20, precisely when they occur. In this example, the input to the system $\theta$ is time and y is the output of the system.
• Figure 4: DiHRNN structure of the Bergman Minimal Model