Online Distribution Shift Detection via Recency Prediction

TL;DR

This work tackles online distribution shift detection for robotics, addressing the need for rapid, streaming, high-dimensional shift awareness with formal guarantees. It introduces a self-supervised, end-to-end framework that learns a recency-predictor and builds an exponential martingale to issue warnings when non-exchangeability arises, achieving an $\epsilon$-sound online guarantee and a false positive rate capped by $1/C$. The approach yields up to 11x faster shift detection in photorealistic simulations and hardware experiments (X-Plane taxiing and free-flyer docking) while maintaining a strict $1\%$ false alarm bound, demonstrating practical viability for safety-critical robotics. The combination of deep recency prediction with martingale theory enables rapid, scalable monitoring of high-dimensional inputs like images, and the results suggest this method can meaningfully enhance reliability in deployed learning-enabled systems.

Abstract

When deploying modern machine learning-enabled robotic systems in high-stakes applications, detecting distribution shift is critical. However, most existing methods for detecting distribution shift are not well-suited to robotics settings, where data often arrives in a streaming fashion and may be very high-dimensional. In this work, we present an online method for detecting distribution shift with guarantees on the false positive rate - i.e., when there is no distribution shift, our system is very unlikely (with probability $< ε$) to falsely issue an alert; any alerts that are issued should therefore be heeded. Our method is specifically designed for efficient detection even with high dimensional data, and it empirically achieves up to 11x faster detection on realistic robotics settings compared to prior work while maintaining a low false negative rate in practice (whenever there is a distribution shift in our experiments, our method indeed emits an alert). We demonstrate our approach in both simulation and hardware for a visual servoing task, and show that our method indeed issues an alert before a failure occurs.

Figures (9)

• Figure 1: Illustration of our problem setting and high-level approach. Learning enabled robotics systems, such as a vision-based aircraft controller, are trained on data from a finite set of environments (e.g. images taken in the morning and afternoon). When deployed, these systems may operate in distribution-shifted conditions, resulting in erroneous predictions on out-of-distribution data. To improve safety, we design a warning system that can detect distribution shifts in a streaming fashion with a guaranteed false positive rate.
• Figure 2: Martingale values for our method (blue), the CM method (orange), and the modified CM method CM-FV (green). The distribution shifts gradually over the course of the day, and an alert is issued when the martingales reach 100. In this example, our method issues an alert at time step 13, CM-FV issues an alert at time step 43, and CM issues an alert at time step 143.
• Figure 3: Sample images generated with the X-Plane 11 flight simulator, with (\ref{['fig:camera_correct']}) a standard camera angle, and (\ref{['fig:camera_perturbed']}) a perturbed camera angle. (\ref{['fig:xplane_camera_results']}) shows martingale values for our method with (blue) and without (orange) distribution shift. With a distribution shift, the martingale grows rapidly, but without one, the martingale does not grow. We empirically observe both FNR = 0 and FPR = 0.
• Figure 4: Gradual degradation of the visual target, after (\ref{['fig:0episodes']}) 0 episodes, (\ref{['fig:5episodes']}) 5 episodes, (\ref{['fig:21episodes']}) 21 episodes, when our method issues an alert, and (\ref{['fig:30episodes']}) 30 episodes, when the robot fails to navigate to the target.
• Figure 5: (\ref{['fig:freeflyer_setup']}) Hardware setup with camera mounted on the side of the mobile robot. (\ref{['fig:freeflyer_target']}) The visual servoing target that the robot navigates to.

Theorems & Definitions (11)

• Definition 1: Martingale
• Proposition 1: Doob's Inequality
• Definition 2: Exchangeability
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 1
• proof
• Lemma 2