Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulation

Huanyu Li, Kun Lei, Sheng Zang, Kaizhe Hu, Yongyuan Liang, Bo An, Xiaoli Li, Huazhe Xu

TL;DR

This work tackles IR Failures during real-world RL by proposing FARL, a failure-aware offline-to-online reinforcement learning framework augmented with a world-model-based safety critic and a recovery policy trained offline. It couples offline pre-training of a task policy, recovery policy, and world model with online fine-tuning that uses action correction to avoid high-risk transitions, guided by a horizon-based constraint $C_H^{\pi} \leq \varepsilon_{safe}$. The FailueBench simulation suite and real-robot experiments on a Franka Panda demonstrate substantial reductions in IR Failures (e.g., up to ~74% decrease) while maintaining or improving task performance and generalization. The work provides theoretical guarantees through an action-correction analysis and highlights practical implications for safer and more reliable real-world robotic learning.

Abstract

Post-training algorithms based on deep reinforcement learning can push the limits of robotic models for specific objectives, such as generalizability, accuracy, and robustness. However, Intervention-requiring Failures (IR Failures) (e.g., a robot spilling water or breaking fragile glass) during real-world exploration happen inevitably, hindering the practical deployment of such a paradigm. To tackle this, we introduce Failure-Aware Offline-to-Online Reinforcement Learning (FARL), a new paradigm minimizing failures during real-world reinforcement learning. We create FailureBench, a benchmark that incorporates common failure scenarios requiring human intervention, and propose an algorithm that integrates a world-model-based safety critic and a recovery policy trained offline to prevent failures during online exploration. Extensive simulation and real-world experiments demonstrate the effectiveness of FARL in significantly reducing IR Failures while improving performance and generalization during online reinforcement learning post-training. FARL reduces IR Failures by 73.1% while elevating performance by 11.3% on average during real-world RL post-training. Videos and code are available at https://failure-aware-rl.github.io.

Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulation

TL;DR

This work tackles IR Failures during real-world RL by proposing FARL, a failure-aware offline-to-online reinforcement learning framework augmented with a world-model-based safety critic and a recovery policy trained offline. It couples offline pre-training of a task policy, recovery policy, and world model with online fine-tuning that uses action correction to avoid high-risk transitions, guided by a horizon-based constraint . The FailueBench simulation suite and real-robot experiments on a Franka Panda demonstrate substantial reductions in IR Failures (e.g., up to ~74% decrease) while maintaining or improving task performance and generalization. The work provides theoretical guarantees through an action-correction analysis and highlights practical implications for safer and more reliable real-world robotic learning.

Abstract

Post-training algorithms based on deep reinforcement learning can push the limits of robotic models for specific objectives, such as generalizability, accuracy, and robustness. However, Intervention-requiring Failures (IR Failures) (e.g., a robot spilling water or breaking fragile glass) during real-world exploration happen inevitably, hindering the practical deployment of such a paradigm. To tackle this, we introduce Failure-Aware Offline-to-Online Reinforcement Learning (FARL), a new paradigm minimizing failures during real-world reinforcement learning. We create FailureBench, a benchmark that incorporates common failure scenarios requiring human intervention, and propose an algorithm that integrates a world-model-based safety critic and a recovery policy trained offline to prevent failures during online exploration. Extensive simulation and real-world experiments demonstrate the effectiveness of FARL in significantly reducing IR Failures while improving performance and generalization during online reinforcement learning post-training. FARL reduces IR Failures by 73.1% while elevating performance by 11.3% on average during real-world RL post-training. Videos and code are available at https://failure-aware-rl.github.io.
Paper Structure (21 sections, 1 theorem, 19 equations, 8 figures, 3 tables)

This paper contains 21 sections, 1 theorem, 19 equations, 8 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Problem Statement
  4. Method
  5. Offline Pre-training
  6. Online fine-tuning with recovery
  7. Theoretical Analysis
  8. Preliminaries
  9. Assumptions
  10. Main Result
  11. Discussion
  12. Experiments
  13. FailureBench: Simulation Benchmark for Evaluating Failure-Aware RL
  14. Simulation Experiments
  15. Experimental Setup
  16. ...and 6 more sections

Key Result

Theorem 1

Under Assumptions assum:risk-assum:advantage, the policy improvement from FARL's corrected transitions satisfies: where $p_{risk}(s) = \mathbb{P}_{a \sim \pi_{task}}[a \in \mathcal{A}_{risk}(s)]$ is the probability of sampling risky actions at state $s$.

Figures (8)

  • Figure 1: FARL demonstrated on a Franka Emika Panda robot across three manipulation tasks: (a) pushing fragile objects while avoiding wall collisions, (b) pushing with dynamic obstacle avoidance, (c) soccer with boundary constraints (detailed in Section VI.C.1). FARL predicts potential failures and executes recovery actions, significantly reducing Intervention-requiring Failures during real-world RL while improving task performance.
  • Figure 2: The entire training pipeline consists of two main phases: 1) the offline phase, which involves pre-training the task policy, recovery policy, and world model, and 2) the online phase, during which the task policy is fine-tuned within safe exploration settings.
  • Figure 3: Illustration of the four tasks in our FailureBench. From left to right: Bounded Push, Bounded Soccer, Fragile Push Wall, and Obstructed Push.
  • Figure 4: Comparison of average failure episodes during fine-tuning for Uni-O4 (blue) and our method (red) in FailureBench.
  • Figure 5: Performance comparison between Uni-O4 (blue) and our method (red) in FailureBench. The bars show average return before and after fine-tuning. All returns are normalized relative to an expert script policy's performance (100).
  • ...and 3 more figures

Theorems & Definitions (2)

  • Theorem 1: Action Correction Benefit
  • proof : Proof Sketch