Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning

TL;DR

ARMOR introduces an adaptive round-based multi-task framework for robotic failure detection and open-ended reasoning from video data. It tackles heterogeneous supervision by combining large-scale sparse failure labels with smaller dense reasoning annotations and uses offline imitation plus online refinement. The method achieves state-of-the-art performance on detection and reasoning across diverse datasets, with detection gains up to 30% and reasoning gains up to 100% in LLM-based metrics. The approach enables robust, human-like failure explanations in open-ended scenarios and reduces reliance on exhaustive failure taxonomies.

Abstract

Reasoning about failures is crucial for building reliable and trustworthy robotic systems. Prior approaches either treat failure reasoning as a closed-set classification problem or assume access to ample human annotations. Failures in the real world are typically subtle, combinatorial, and difficult to enumerate, whereas rich reasoning labels are expensive to acquire. We address this problem by introducing ARMOR: Adaptive Round-based Multi-task mOdel for Robotic failure detection and reasoning. We formulate detection and reasoning as a multi-task self-refinement process, where the model iteratively predicts detection outcomes and natural language reasoning conditioned on past outputs. During training, ARMOR learns from heterogeneous supervision - large-scale sparse binary labels and small-scale rich reasoning annotations - optimized via a combination of offline and online imitation learning. At inference time, ARMOR generates multiple refinement trajectories and selects the most confident prediction via a self-certainty metric. Experiments across diverse environments show that ARMOR achieves state-of-the-art performance by improving over the previous approaches by up to 30% on failure detection rate and up to 100% in reasoning measured through LLM fuzzy match score, demonstrating robustness to heterogeneous supervision and open-ended reasoning beyond predefined failure modes. We provide dditional visualizations on our website: https://sites.google.com/utexas.edu/armor

Figures (6)

• Figure 1: We present ARMOR: adaptive round-based multi-task model for robotic failure detection and reasoning. Prior approaches reduce failure reasoning to closed-set classification of pre-defined modes (e.g., “spillage” or “unstable grasp”). In contrast, ARMOR performs open-ended, iterative refinement by jointly refining its detection and reasoning predictions. This enables accurate detection with nuanced and human-like reasoning that capture real-world failures beyond fixed categories.
• Figure 2: Overview of ARMOR. (a) Our failure data consist of heterogeneous supervision, with large-scale binary detection labels and scarce free-form reasoning labels. (b) A vision-language model (VLM) with multi-task heads jointly predicts detection $l$ via a classification head and reasoning $e$ via a language decoder, trained with binary cross-entropy (BCE) and next-token prediction (NTP) losses. (c) We fine-tune the VLM with both offline imitation and online refinement: the model conditions on dataset labels ($l, e$) or its prior predictions ($\hat{l}, \hat{e}$) as well as auxiliary prompts $p$ to generate a new round of outputs. In online refinement, this process is repeated $T$ times. The model's predictions (denoted by $\hat{l}$ and $\hat{e}$ in purple rectangles) are supervised with available labels. (d) At inference, ARMOR performs iterative self-refinement, rolling out multiple stochastic trajectories. We select best prediction $m^\star$ from the final predictions with the lowest entropy score.
• Figure 3: Refinement analysis in ARMOR. (a) Effect of refinement rounds on entropy and performance: detection and reasoning entropy decrease steadily across rounds, while combined plots show that refinement improves both detection accuracy and reasoning quality (values computed over 300 test datapoints). (b) Example of ARMOR’s multi-round refinement process, where predictions are iteratively updated to improve the quality and consistency between detection and reasoning outputs.
• Figure 4: ARMOR model architecture. We select the intermediate layer representation from the LM decoder of Qwen2.5-VL bai2025qwen2 and attach a classification head for detection, while using the original LM decoder for reasoning. Conditioning prompts describe the previous outputs for each task, enabling iterative multi-task refinement.
• Figure 5: Hockey-stick task from RLBench Multi-view frames (front, wrist, overhead) across timesteps showing the robot attempting to pick up a hockey stick.