Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models

Alexander W. Goodall, Francesco Belardinelli

TL;DR

The paper tackles the challenge of provable safety in reinforcement learning for unknown, non-linear, continuous dynamical systems by introducing recovery-based shielding that combines a learned policy with a pre-computed backup controller. Safety is enforced through analytic GP-based uncertainty sets that predict potential constraint violations, enabling on-the-fly recovery to a verified invariant region without relying on sampling. A formal $\\epsilon$-safe guarantee is established under standard regularity assumptions, with safety maintained throughout learning and deployment once GP calibration is achieved. The approach achieves strong safety (zero violations) while delivering competitive rewards across a suite of continuous control tasks, and demonstrates scalability to higher-dimensional systems like Hopper-v5. Overall, the work offers a principled, data-efficient framework for safe RL that operates without requiring full knowledge of the environment dynamics or exhaustive sampling for safety verification.

Abstract

Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications. In this paper, we introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems. The proposed approach integrates a backup policy (shield) with the RL agent, leveraging Gaussian process (GP) based uncertainty quantification to predict potential violations of safety constraints, dynamically recovering to safe trajectories only when necessary. Experience gathered by the 'shielded' agent is used to construct the GP models, with policy optimization via internal model-based sampling - enabling unrestricted exploration and sample efficient learning, without compromising safety. Empirically our approach demonstrates strong performance and strict safety-compliance on a suite of continuous control environments.

Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models

TL;DR

The paper tackles the challenge of provable safety in reinforcement learning for unknown, non-linear, continuous dynamical systems by introducing recovery-based shielding that combines a learned policy with a pre-computed backup controller. Safety is enforced through analytic GP-based uncertainty sets that predict potential constraint violations, enabling on-the-fly recovery to a verified invariant region without relying on sampling. A formal -safe guarantee is established under standard regularity assumptions, with safety maintained throughout learning and deployment once GP calibration is achieved. The approach achieves strong safety (zero violations) while delivering competitive rewards across a suite of continuous control tasks, and demonstrates scalability to higher-dimensional systems like Hopper-v5. Overall, the work offers a principled, data-efficient framework for safe RL that operates without requiring full knowledge of the environment dynamics or exhaustive sampling for safety verification.

Abstract

Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications. In this paper, we introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems. The proposed approach integrates a backup policy (shield) with the RL agent, leveraging Gaussian process (GP) based uncertainty quantification to predict potential violations of safety constraints, dynamically recovering to safe trajectories only when necessary. Experience gathered by the 'shielded' agent is used to construct the GP models, with policy optimization via internal model-based sampling - enabling unrestricted exploration and sample efficient learning, without compromising safety. Empirically our approach demonstrates strong performance and strict safety-compliance on a suite of continuous control environments.
Paper Structure (55 sections, 4 theorems, 46 equations, 9 figures, 11 tables, 2 algorithms)

This paper contains 55 sections, 4 theorems, 46 equations, 9 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Preliminaries
  4. Problem setup
  5. Safety semantics
  6. Safe RL objective.
  7. Dynamics modelling
  8. Recovery-based shielding
  9. Overview
  10. Control invariant sets
  11. Backup policy.
  12. Recoverable states
  13. GP uncertainty prediction
  14. Prediction at uncertain inputs
  15. Uncertainty propagation.
  16. ...and 40 more sections

Key Result

Theorem 3.1

Assume: (i) $\mathcal{X}_{\text{inv}}$ is an invariant set for $\pi_{\text{backup}}$, (ii) $\mathcal{X}_0 \subseteq \mathcal{X}_{\text{inv}} \subseteq \mathcal{X}_{\text{safe}}$ and (iii) $\sum^{T}_{t=0}\epsilon_t = \epsilon$, then, $\pi_{\text{shield}}$ is $\epsilon$-safe for the time horizon $T \i

Figures (9)

  • Figure 1: $99.99\%$-CI uncertainty sets for recoverable (blue) and irrecoverable (red) trajectories under the backup policy for cartpole (i)/(ii) environment.
  • Figure 2: Learning curves (log scale on the x-axis).
  • Figure 3: Learning curves (reward and safety probability) for A2C-Shield and A2C-Eval.
  • Figure 4: Overrides (per episode) for A2C-Shield
  • Figure 5: Learning curves (reward and safety probability) for CPO and PPO-Lag.
  • ...and 4 more figures

Theorems & Definitions (9)

  • Definition 2.1: $\epsilon$-safe policy
  • Definition 3.1: Recoverable state
  • Definition 3.2: $\epsilon$-Recoverable state
  • Theorem 3.1: $\epsilon$-safe policy
  • Remark
  • Theorem 4.1
  • Remark
  • Theorem \ref{thm:epsilonsafe} (restated): $\epsilon$-Safe Policy
  • Theorem \ref{thm:epsilonsafe} (restated)