Failure Prediction from Limited Hardware Demonstrations

TL;DR

This work presents a novel three-step methodology for discovering failures that occur in the true system by using a combination of a limited number of demonstrations from the true system and the failure information processed through sampling-based testing of a model dynamical system.

Abstract

Prediction of failures in real-world robotic systems either requires accurate model information or extensive testing. Partial knowledge of the system model makes simulation-based failure prediction unreliable. Moreover, obtaining such demonstrations is expensive, and could potentially be risky for the robotic system to repeatedly fail during data collection. This work presents a novel three-step methodology for discovering failures that occur in the true system by using a combination of a limited number of demonstrations from the true system and the failure information processed through sampling-based testing of a model dynamical system. Given a limited budget $N$ of demonstrations from true system and a model dynamics (with potentially large modeling errors), the proposed methodology comprises of a) exhaustive simulations for discovering algorithmic failures using the model dynamics; b) design of initial $N_1$ demonstrations of the true system using Bayesian inference to learn a Gaussian process regression (GPR)-based failure predictor; and c) iterative $N - N_1$ demonstrations of the true system for updating the failure predictor. To illustrate the efficacy of the proposed methodology, we consider: a) the failure discovery for the task of pushing a T block to a fixed target region with UR3E collaborative robot arm using a diffusion policy; and b) the failure discovery for an F1-Tenth racing car tracking a given raceline under an LQR control policy.

Figures (7)

• Figure 1: The proposed methodology constitutes a) discovering failures using model information; b) design of initial demonstrations to learn true system failures using Bayesian inference; and c) sequential demonstration from low predicted-risk regions for GPR-based risk prediction update.
• Figure 2: Venn Diagram showing different sets defined in Section. \ref{['sec:methodology']}. Here, $\mathcal{Z}_{\text{fail}}(f)$ and $\mathcal{Z}_{\text{noise}}$ denote the set of failures that can be discovered using model dynamics (Fig. \ref{['fig:method']}-a. (i) and (ii) respectively), and $\mathcal{Z}_{\text{real}}$ is estimated using demonstrations from the true system (Fig. \ref{['fig:method']}-b. and c.)
• Figure 3: The simulation (model dynamics) environment and the hardware (actual dynamics) setup for the Push-T (top figures) and F1-Tenth (bottom figures) examples.
• Figure 4: Risk predictions for Push-T example from GPR trained only simulation data (left) and using the proposed methodology (right).
• Figure 5: Model prediction for Push-T on $20$ data points where the predicted risk by the learned model $\phi_\theta$. The demonstration on true system marked as 'Ground Truth' illustrates the prediction to be accurate.

Theorems & Definitions (1)

• Remark 1