Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models

TL;DR

An automated failure synthesis pipeline generates diverse planning and execution failures to train Vision-Language Models for robotic failure detection. The Guardian model, a reasoning VLM built on multi-view inputs and chain-of-thought traces, achieves state-of-the-art accuracy on new benchmarks RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail and generalizes to real-world data without fine-tuning. The approach provides fine-grained failure categorization and reasoning, and when integrated into a manipulation framework, substantially improves task success in both simulation and real robots. This work closes the data gap for failure detection and offers a practical, plug-and-play verifier for robust robotic manipulation.

Abstract

Robust robotic manipulation requires reliable failure detection and recovery. Although current Vision-Language Models (VLMs) show promise, their accuracy and generalization are limited by the scarcity of failure data. To address this data gap, we propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures. This method produces not only binary classification labels but also fine-grained failure categories and step-by-step reasoning traces in both simulation and the real world. With it, we construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail, substantially expanding the diversity and scale of existing failure datasets. We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection. Guardian achieves state-of-the-art performance on both existing and newly introduced benchmarks. It also effectively improves task success rates when integrated into a state-of-the-art manipulation system in simulation and real robots, demonstrating the impact of our generated failure data. Code, Data, and Models available at https://www.di.ens.fr/willow/research/guardian/.

Figures (9)

• Figure 1: Illustration of our Guardian model - a VLM fine-tuned on our constructed failure datasets. It detects planning failures (top) and execution failures (bottom) in robotic manipulation.
• Figure 2: Failure Data Generation Pipeline. We introduce a novel generation pipeline generating failure cases both online in simulation (RLBench), and offline on the real-world dataset (BridgeDataV2). For each positive example, given its correct plan and successful trajectory, we generate a corresponding incorrect plan and unsuccessful trajectory.
• Figure 3: Training examples. Guardian is trained on reasoning traces produced by our automatic data generation pipeline.
• Figure 4: Failure mode distributions in real executions and our constructed data.
• Figure 5: Left: Overview of the Guardian model architecture. Right: Integration of Guardian model into a robot manipulation pipeline for planning and execution verification.