Sample-Efficient Safety Assurances using Conformal Prediction

TL;DR

The paper tackles the challenge of guaranteeing safety in robotics warning systems with limited data. It adapts Mondrian conformal prediction to a robotics setting, enabling provable $\epsilon$-safe false negative rates using as few as $1/\epsilon$ unsafe examples, under a single exchangeability assumption. The approach is instantiated and validated on driver alert and robotic grasping tasks, demonstrating tight control of the false negative rate while maintaining a low false positive rate, and offering favorable sample-efficiency relative to traditional PAC learning. The work highlights the practical impact of provable safety guarantees in time-series, sequential-decision robotics, and sets the stage for future extensions to non-exchangeable data and conditional safety guarantees.

Abstract

When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial. Early warning systems can provide alerts when an unsafe situation is imminent (in the absence of corrective action). To reliably improve safety, these warning systems should have a provable false negative rate; i.e. of the situations that are unsafe, fewer than $ε$ will occur without an alert. In this work, we present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics, in order to tune warning systems to provably achieve an $ε$ false negative rate using as few as $1/ε$ data points. We apply our framework to a driver warning system and a robotic grasping application, and empirically demonstrate guaranteed false negative rate while also observing low false detection (positive) rate.

Figures (13)

• Figure 1: We design a warning system that achieves a provable false negative rate sample efficiently. Among the situations that are dangerous (i.e. lead to an unsafe future situation in the absence of corrective action), fewer than $\epsilon$ occur without an alert.
• Figure 2: Overview of our problem setup: in this simplified example, there is an ego-agent (shown in blue), and an external agent (shown in red), whose position we would like to predict. $X$ represents the current location of the external agent, $Y$ represents the predicted future location of the external agent, and $Z$ represents the true future location of the external agent. $f_0$ represents the distance threshold at which two cars are considered too close together (i.e. the situation is considered unsafe), and $f$ and $g$ are both the distance to the nearest car.
• Figure 3: Given the simulation or model output $Y$, the warning function $w(Y)$ either decides to issue a warning ($w(Y) = 1$), or not ($w(Y) = 0$).
• Figure 4: Whenever the true safety score $f(Z)$ is below $f_0$ (i.e. the red car and the blue car will indeed be too close together), the warning system should issue a warning ($w(Y) = 1$) with at least $1 - \epsilon$ probability.
• Figure 5: Even in the limit of infinite validation data, the best false positive rate achievable (for a given $\epsilon$-safety level) is determined by the distribution of the safe samples and the unsafe samples under the surrogate safety score function $g$.

Theorems & Definitions (3)

• Definition 1
• Proposition 1
• Proposition 2