RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction

TL;DR

The results show that the RoboFAC framework effectively handles robotic failures and assists the VLA model in recovering from failures.

Abstract

Vision-Language-Action (VLA) models have recently advanced robotic manipulation by translating natural-language instructions and image information into sequential control actions. However, these models often underperform in open-world scenarios, as they are predominantly trained on successful expert demonstrations and exhibit a limited capacity for failure recovery. In this work, we present a Robotic Failure Analysis and Correction (RoboFAC) framework to address this issue. Firstly, we construct RoboFAC dataset comprising 9,440 erroneous manipulation trajectories and 78,623 QA pairs across 16 diverse tasks and 53 scenes in both simulation and real-world environments. Leveraging our dataset, we develop RoboFAC model, which is capable of Task Understanding, Failure Analysis and Failure Correction. Experimental results demonstrate that the RoboFAC model outperforms GPT-4o by 34.1% on our evaluation benchmark. Furthermore, we integrate the RoboFAC model into a real-world VLA control pipeline as an external supervision providing correction instructions, yielding a 29.1% relative improvement on average on four real-world tasks. The results show that our RoboFAC framework effectively handles robotic failures and assists the VLA model in recovering from failures.

Figures (7)

• Figure 1: Overview of RoboFAC dataset. Left: The RoboFAC dataset features both task diversity and visual diversity, encompassing tasks of varying complexity, real-world tasks, and various of backgrounds and camera viewpoints. We provide detailed video question-answer annotations for eight distinct question types. Right: A detailed visual illustration of the six failure taxonomies.
• Figure 2: Statistics of the RoboFAC Dataset. Left: Categories of robotic tasks in the RoboFAC dataset. (Lh. Task: Long-horizon task, Mh. Task: Medium-horizon task, Sh. Task: Short-horizon Task, Dy. Task: Dynamic Task) Top Right: Distribution of video counts by duration interval. Bottom Right: Average duration of each task.
• Figure 3: Overview of our RoboFAC framework. Top: The pipeline of constructing the RoboFAC dataset. Bottom-left: We build our RoboFAC model by fine-tuning Qwen2.5-VL model. The RoboFAC model can perform Task Understanding, Failure analysis and Failure correction. Bottom-right: We deploy RoboFAC model on real-world VLA control tasks, and it effectively helps the VLA recover from failure.
• Figure 4: Scores for different dimensions on RoboFAC Benchmark Left: Performance on different question dimensions for simulation dataset. Top Right: Performance on different question dimensions for real world dataset. Bottom Right: Performance on different real world tasks.
• Figure 5: Qualitative comparison of failure explanations generated by RoboFAC-7B and GPT-4o across different tasks.