SAFE: Multitask Failure Detection for Vision-Language-Action Models

TL;DR

This work tackles the safety challenge of generalist Vision-Language-Action models by introducing SAFE, a multitask failure detector that operates on a VLA's internal latent features. SAFE trains on successful and failed rollouts across multiple tasks and uses a lightweight backbone (MLP or LSTM) to produce a per-timestep failure score $s_t$, with a time-varying threshold $\delta_t$ calibrated via functional conformal prediction (upper_t = $\mu_t + h_t$). By leveraging the observed separation between success and failure in the VLA feature space, SAFE generalizes to unseen tasks and architectures, achieving state-of-the-art ROC-AUC on simulation and real-world benchmarks (OpenVLA, $\pi_0$, $\pi_0$-FAST) while maintaining low runtime overhead (≈0.7 ms) and a modest parameter footprint (~2.3M). The approach offers principled, calibrated failure detection that supports timely intervention, backtracking, or human oversight, advancing the practical deployment of robust, multitask robotic policies.

Abstract

While vision-language-action models (VLAs) have shown promising robotic behaviors across a diverse set of manipulation tasks, they achieve limited success rates when deployed on novel tasks out of the box. To allow these policies to safely interact with their environments, we need a failure detector that gives a timely alert such that the robot can stop, backtrack, or ask for help. However, existing failure detectors are trained and tested only on one or a few specific tasks, while generalist VLAs require the detector to generalize and detect failures also in unseen tasks and novel environments. In this paper, we introduce the multitask failure detection problem and propose SAFE, a failure detector for generalist robot policies such as VLAs. We analyze the VLA feature space and find that VLAs have sufficient high-level knowledge about task success and failure, which is generic across different tasks. Based on this insight, we design SAFE to learn from VLA internal features and predict a single scalar indicating the likelihood of task failure. SAFE is trained on both successful and failed rollouts and is evaluated on unseen tasks. SAFE is compatible with different policy architectures. We test it on OpenVLA, $π_0$, and $π_0$-FAST in both simulated and real-world environments extensively. We compare SAFE with diverse baselines and show that SAFE achieves state-of-the-art failure detection performance and the best trade-off between accuracy and detection time using conformal prediction. More qualitative results and code can be found at the project webpage: https://vla-safe.github.io/

Figures (10)

• Figure 1: The internal features of a VLA capture high-level information about task success and failure. When the VLA is failing, the features, even those from different tasks, fall into the same "failure zone". This motivates SAFE, an efficient multitask failure detector that is based on VLA internal features and can generalize to unseen tasks. Plot (a) visualizes the latent features of $\pi_0$-FAST on LIBERO-10 liu2023libero using t-SNE tsne. For successful rollouts, features are colored in blue. For failed rollouts, features follow a blue-to-red gradient based on timestep progression, with red corresponding to later timesteps that often coincide with failure. Plot (b) visualizes the same set of t-SNE features, colored by task ID. In (c), we show two example rollouts over time and mark their corresponding projected features in (a) and (b).
• Figure 2: The proposed failure detector, SAFE, has three major components: (1) SAFE extracts the latent feature from the last layer of a VLA model; (2) SAFE sequentially processes the latent feature and predicts a failure score, using an MLP or an LSTM backbone; and (3) SAFE determines a time-varying threshold using functional conformal prediction (CP) on a hold-out calibration set. If the predicted score exceeds the threshold during testing, SAFE confidently detects a failure.
• Figure 3: Illustration of real-world experiment setup (left) and example rollouts collected (right).
• Figure 4: In all simulation experiments, the proposed SAFE-LSTM and SAFE-MLP perform better than or on par with the best baselines. The plots show the variation of balanced accuracy (bal-acc) with respect to average detection time (T-det) on $\mathcal{D}_\text{eval-unseen}$, under different significance levels $\alpha$ used for functional CP. Good failure detection methods should detect policy failures both accurately (high bal-acc) and proactively (lower T-det), and thus place curves towards the top left in each plot. Note that baselines in gray require multiple action samples.
• Figure 5: Failures detected by SAFE-LSTM align well with the actual robot failures, as shown in the corresponding camera observations from simulation experiments. The blue-shaded areas show the functional CP band $C_\alpha$. Once failure scores exceed $C_\alpha$, a failure flag is raised. In (a), the $\pi_0$-FAST policy misses the insertion, and its actions become unstable after that. In (b) and (c), OpenVLA and $\pi_0^*$ miss the grasp but still proceed to the placing action, causing a failure detection. Note that these tasks are not seen when training SAFE-LSTM.