Recover: A Neuro-Symbolic Framework for Failure Detection and Recovery

TL;DR

This work presents Recover, a neuro-symbolic framework for online failure detection and recovery in robotic task execution. By integrating an ontology-based symbolic model with LLM-based planners, Recover detects failures in real time using rule-based reasoning on multimodal observations and generates recoveries that are mapped to executable actions, reducing the need for costly scene resets. The OntoThor ontology grounds the AI2Thor kitchen environment, enabling detailed failure taxonomy and recovery strategies. Experimental results show strong online failure detection, substantial recovery rates, safety assurance, and clear cost advantages over purely LLM-based baselines, highlighting the practical value of combining symbolic reasoning with learned planning in dynamic environments.

Abstract

Recognizing failures during task execution and implementing recovery procedures is challenging in robotics. Traditional approaches rely on the availability of extensive data or a tight set of constraints, while more recent approaches leverage large language models (LLMs) to verify task steps and replan accordingly. However, these methods often operate offline, necessitating scene resets and incurring in high costs. This paper introduces Recover, a neuro-symbolic framework for online failure identification and recovery. By integrating ontologies, logical rules, and LLM-based planners, Recover exploits symbolic information to enhance the ability of LLMs to generate recovery plans and also to decrease the associated costs. In order to demonstrate the capabilities of our method in a simulated kitchen environment, we introduce OntoThor, an ontology describing the AI2Thor simulator setting. Empirical evaluation shows that OntoThor's logical rules accurately detect all failures in the analyzed tasks, and that Recover considerably outperforms, for both failure detection and recovery, a baseline method reliant solely on LLMs.

Figures (3)

• Figure 1: Overview of the Recover framework. Starting with a plan the robot executes one action at a time over the environment. The outcome of each action conveyed through multi-modality information (audio and video) is then processed and converted into an audio label and a scene-graph. These are stored within the ontology and provided as input to the sub-goal verifier, which classifies the action execution as either a failure or a success. If the action is successful, the robot proceeds with the next step. In the event of failure, reasoning module will use the failure information to extract the recovery strategy from the ontology and will supply it to the LLM-planner. Subsequently, the LLM-planner generates a new plan to recover from the failure and accomplish the task. Figure legend: Blue elements represent model-based components, while orange elements denote policy-based components. Sharp-cornered shapes indicate input/output elements, whereas round-cornered shapes signify procedures. Dashed lines correspond to input sources, while solid lines indicate the procedural loop.
• Figure 2: Example of the information flow during plan execution, depicting the alternation of two event types: observation events (in green) and action events (in pink). Each action event can be associated with a sound recorded by the robot during the action's execution. Each observation event is linked to a scene-graph describing the frame captured by the robot. In the scene graph, each node represent an object identified in the scene and each edge represent the relation between two objects. The colors of the nodes in the scene graph correspond to those in the segmentation image (the adjacent image) where the objects have been identified.
• Figure 3: Failures taxonomy in OntoThor