Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols

TL;DR

ViFailback introduces a visual-symbol–driven framework to diagnose, correct, and learn from robotic manipulation failures using real-world data. It crowdsources texture-rich annotations via visual symbols and text, builds ViFailback-8B through fine-tuning on 58,126 VQA pairs, and offers ViFailback-Bench with Lite and Hard settings to evaluate failure reasoning. The approach yields significant gains over baselines and, when deployed as an external supervisor, improves real-world policy recovery by around 22%. The work demonstrates that visual-symbol guidance can provide actionable supervision bridging real-time correction with policy learning for robust VLA-based manipulation. Overall, ViFailback enables scalable, interpretable learning from failures in real-world robotic systems.

Abstract

Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance. Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the dataset, we establish ViFailback-Bench, a benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of Vision-Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation. To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance. Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures. Project Website: https://x1nyuzhou.github.io/vifailback.github.io/

Figures (28)

• Figure 1: The pipeline leverages real-world failure data to build a dataset and train ViFailback-8B for failure diagnosis and correction.
• Figure 1: Examples of individual visual symbols and their combination.Top and Middle Rows: Instances of individual visual symbols. Bottom Row: Instances of their usage in combination.
• Figure 2: Overview of ViFailback Framework.Left: We collect real-world manipulation trajectories via teleoperation and policy rollout, then use our high-efficiency, visual-symbol-based annotation framework to generate VQA pairs for the dataset. Middle: Our dataset comprises 58,126 VQA pairs from 5,202 real-world trajectories. We extract ViFailback-Bench (Lite and Hard) from this dataset to evaluate VLM failure diagnosis and correction capabilities. Right: We fine-tune Qwen3-VL-8B on our VQA pairs to obtain ViFailback-8B. This model is deployed as an external supervisor to assist the policy in recovering from failures.
• Figure 2: Full list of our designed tasks. (Continued on next page)
• Figure 3: An overview of ViFailback-Bench.Left: The Lite benchmark uses closed-ended VQA to test VLM failure diagnosis (e.g., detection, localization) and low-level correction guidance. Right: The Hard benchmark uses open-ended VQA to test failure reason and high-level/CoT-based guidance.