Learning to Recover from Plan Execution Errors during Robot Manipulation: A Neuro-symbolic Approach

TL;DR

This work tackles the challenge of detecting and recovering from execution errors in long-horizon robot manipulation. It introduces a neuro-symbolic framework that uses a dense scene-graph representation, consisting of a scene-graph encoder $H_\psi$, a scene-graph predictor $T^{ideal}_\theta$, and a scene-graph discriminator $K_\phi$, to imagine nominal plan execution and identify discrepancies without annotated failure data. When errors are detected, a neuro-symbolic search crafts a recovery plan that reuses parts of the original plan and targets sub-goals $S_{t_l}$, with an anytime variant allowing adjustable planning budgets. The approach demonstrates superior recovery efficiency and accuracy compared to RL baselines and full re-planning strategies in a simulated PyBullet environment with varied error types. Overall, the method offers a practical, self-supervised pipeline for robust long-horizon manipulation by coupling perception, prediction, and discrepancy-guided planning.

Abstract

Automatically detecting and recovering from failures is an important but challenging problem for autonomous robots. Most of the recent work on learning to plan from demonstrations lacks the ability to detect and recover from errors in the absence of an explicit state representation and/or a (sub-) goal check function. We propose an approach (blending learning with symbolic search) for automated error discovery and recovery, without needing annotated data of failures. Central to our approach is a neuro-symbolic state representation, in the form of dense scene graph, structured based on the objects present within the environment. This enables efficient learning of the transition function and a discriminator that not only identifies failures but also localizes them facilitating fast re-planning via computation of heuristic distance function. We also present an anytime version of our algorithm, where instead of recovering to the last correct state, we search for a sub-goal in the original plan minimizing the total distance to the goal given a re-planning budget. Experiments on a physics simulator with a variety of simulated failures show the effectiveness of our approach compared to existing baselines, both in terms of efficiency as well as accuracy of our recovery mechanism.

Figures (6)

• Figure 1: Error Recovery Problem. The diagram illustrates an error resulting from the robot colliding with a partially constructed tower. The states along the blue line represent the expected progression in the plan execution devoid of errors. However, the collision results in a significant deviation from this expected path, requiring error recovery planning.
• Figure 2: (a) Model Overview: The learnt scene graph predictor $\mathcal{T}^{ideal}_\theta$ is used to imagine the effect of an action execution, which is then compared with the actual scene graph to detect errors. Significant deviations are marked as error by Discriminator $K_\phi$, and the error recovery module will be triggered. (b) Scene graph Predictor: The architecture of scene-graph extractor and predictor (c) Discriminator: Architecture of the discriminator.
• Figure 3: Recovery Plan Success with Increasing Errors. Experiments analyse the recovery accuracy and time to generate a recovery plan with an increasing number of introduced errors. Note that the length of the optimal recovery plan measures the complexity of the errors.
• Figure 4: Scalability of Recovery Plan Generation Approach. Experiments analyse the reachability of the final goal in long-horizon plans (top) and with an increasing number of objects (bottom). The proposed recovery method (in red) performs effectively for long-horizon plans and can scale to larger scenes as compared to the baselines.
• Figure 5: Demonstration with a Franka Emika Robot in PyBullet Physics engine. Detection of errors during plan execution and generation of recovery plans, ultimately attaining the intended goal. Errors include (a) unexpected collisions between objects and the robot, (b) grasping failure and dropping of the block during arm movement and (c) unanticipated actions by a (cooperative) human aiding one step in plan progress. Abbreviations: Y=Yellow, R=Red, G=Green, B=Blue, C=Cyan, W=White, M=Magenta, D=Dice, C=Cube, L=Lego.