Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to be Safe: Deep RL with a Safety Critic

Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, Chelsea Finn

TL;DR

This work tackles the challenge of safe reinforcement learning by learning a task-agnostic safety critic (SQRL) that predicts future failures and constrains policy updates. It introduces a two-phase approach: pre-train a safety critic in a safety-friendly setting and then safely fine-tune on a target task under an epsilon_safe constraint. Empirical results across 2D navigation, quadruped locomotion, and dexterous manipulation show SQRL yields substantially fewer safety incidents and faster, more stable learning than standard RL and prior safe RL methods. The method provides a practical pathway to safer, more reliable RL systems with transfer capabilities for real-world deployment.

Abstract

Safety is an essential component for deploying reinforcement learning (RL) algorithms in real-world scenarios, and is critical during the learning process itself. A natural first approach toward safe RL is to manually specify constraints on the policy's behavior. However, just as learning has enabled progress in large-scale development of AI systems, learning safety specifications may also be necessary to ensure safety in messy open-world environments where manual safety specifications cannot scale. Akin to how humans learn incrementally starting in child-safe environments, we propose to learn how to be safe in one set of tasks and environments, and then use that learned intuition to constrain future behaviors when learning new, modified tasks. We empirically study this form of safety-constrained transfer learning in three challenging domains: simulated navigation, quadruped locomotion, and dexterous in-hand manipulation. In comparison to standard deep RL techniques and prior approaches to safe RL, we find that our method enables the learning of new tasks and in new environments with both substantially fewer safety incidents, such as falling or dropping an object, and faster, more stable learning. This suggests a path forward not only for safer RL systems, but also for more effective RL systems.

Learning to be Safe: Deep RL with a Safety Critic

TL;DR

This work tackles the challenge of safe reinforcement learning by learning a task-agnostic safety critic (SQRL) that predicts future failures and constrains policy updates. It introduces a two-phase approach: pre-train a safety critic in a safety-friendly setting and then safely fine-tune on a target task under an epsilon_safe constraint. Empirical results across 2D navigation, quadruped locomotion, and dexterous manipulation show SQRL yields substantially fewer safety incidents and faster, more stable learning than standard RL and prior safe RL methods. The method provides a practical pathway to safer, more reliable RL systems with transfer capabilities for real-world deployment.

Abstract

Safety is an essential component for deploying reinforcement learning (RL) algorithms in real-world scenarios, and is critical during the learning process itself. A natural first approach toward safe RL is to manually specify constraints on the policy's behavior. However, just as learning has enabled progress in large-scale development of AI systems, learning safety specifications may also be necessary to ensure safety in messy open-world environments where manual safety specifications cannot scale. Akin to how humans learn incrementally starting in child-safe environments, we propose to learn how to be safe in one set of tasks and environments, and then use that learned intuition to constrain future behaviors when learning new, modified tasks. We empirically study this form of safety-constrained transfer learning in three challenging domains: simulated navigation, quadruped locomotion, and dexterous in-hand manipulation. In comparison to standard deep RL techniques and prior approaches to safe RL, we find that our method enables the learning of new tasks and in new environments with both substantially fewer safety incidents, such as falling or dropping an object, and faster, more stable learning. This suggests a path forward not only for safer RL systems, but also for more effective RL systems.

Paper Structure

This paper contains 19 sections, 4 theorems, 17 equations, 11 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Problem Statement
  5. Safety Q-functions for RL
  6. Pre-Training Phase
  7. Fine-Tuning Phase
  8. Analysis
  9. Practical Implementation of SQRL
  10. Experiments
  11. The Performance and Safety of Learning
  12. The Effect of Safety on Learning
  13. Trading Off Risk and Reward with $\epsilon_\text{safe}$
  14. Discussion
  15. Guaranteeing Safety with the Safety Critic
  16. ...and 4 more sections

Key Result

Lemma 1

For any policy $\bar{\pi} \in \Pi_\text{safe}^\epsilon$, the discounted probability of it failing in the future, given by $\mathbb{E}_{s_{t'} \sim \rho_{\bar{\pi}}|s_t}(\gamma_\text{safe}^t {\mathcal{I}}(s_{t'}) \mid t' > t)$, is less than or equal to $\epsilon_\text{safe}$, given a safety-critic $Q

Figures (11)

  • Figure 1: Our approach enables safe RL by pre-training a critic $Q_\text{safe}^{\bar{\pi}}$ that is trained by constraining the actions of a policy $\pi$. This yields safe policy iterates (resulting from optimizing Eq. \ref{['eq:pol-ft']}) that achieve safe episodes throughout fine-tuning, while standard RL approaches will visit unsafe states during adaptation, which results in more failed episodes.
  • Figure 2: Final task performance and cumulative failure rate during fine-tuning. SQRL achieves good performance while also being significantly safer during learning.
  • Figure 3: Environments used in the paper
  • Figure 4: Fine-tuning curves for our method on the Minitaur and Cube target tasks, suggesting that, beyond increasing safety, SQRL's safety-critic leads to more stable and efficient learning.
  • Figure 5: Qualitative results showing different trajectories corresponding to different target safety thresholds.
  • ...and 6 more figures

Theorems & Definitions (6)

  • Lemma 1
  • Theorem 1
  • Theorem 1
  • proof
  • Corollary 1.1
  • proof