A Provable Approach for End-to-End Safe Reinforcement Learning

TL;DR

Provably Lifetime Safe RL (PLS) tackles end-to-end safety in reinforcement learning by first learning a return-conditioned policy offline via safe return-conditioned supervised learning, then safely deploying it and optimally tuning a two-dimensional target return vector $(R,G)$ with Gaussian Process models of actual returns. Theoretical guarantees connect target returns to realized performance through a GP-based bound, ensuring safety with probability at least $1-\Delta$ and enabling near-optimal reward with finite online samples. Empirically, PLS outperforms offline and online baselines on Safety-Gym benchmarks by maintaining safety constraints across tasks while achieving high rewards, and it does so with modest online computation thanks to the low-dimensional target-return optimization. The approach offers a practical, data-efficient route to end-to-end safe RL, though it leaves open the challenge of guaranteeing near-optimal policies in addition to near-optimal target returns.

Abstract

A longstanding goal in safe reinforcement learning (RL) is a method to ensure the safety of a policy throughout the entire process, from learning to operation. However, existing safe RL paradigms inherently struggle to achieve this objective. We propose a method, called Provably Lifetime Safe RL (PLS), that integrates offline safe RL with safe policy deployment to address this challenge. Our proposed method learns a policy offline using return-conditioned supervised learning and then deploys the resulting policy while cautiously optimizing a limited set of parameters, known as target returns, using Gaussian processes (GPs). Theoretically, we justify the use of GPs by analyzing the mathematical relationship between target and actual returns. We then prove that PLS finds near-optimal target returns while guaranteeing safety with high probability. Empirically, we demonstrate that PLS outperforms baselines both in safety and reward performance, thereby achieving the longstanding goal to obtain high rewards while ensuring the safety of a policy throughout the lifetime from learning to operation.

Figures (3)

• Figure 1: A conceptual illustration of PLS. After learning a return-conditioned policy using offline safe RL, PLS optimizes target returns through safe online policy evaluation via Gaussian processes. A key advantage of PLS is that safety is guaranteed at least with high probability in the entire process.
• Figure 2: Relations between target safety cost return $G$ and actual safety cost return $J_g(\pi)$ of pretrained CDT policies (red lines). Blue dotted lines represent $y = x$. Target reward returns are fixed with the reward returns of the best trajectories included in the offline dataset. Observe that CDT policies suffer from unsuccessful misalignment between actual returns and target returns: (a) constraint violation, (b) excessively conservative behavior, and (c) both.
• Figure 3: Experimental results on how our PLS ensures the satisfaction of the safety constraint while obtaining new GP observations. Black dotted lines represent the normalized safety threshold.

Theorems & Definitions (27)

• Theorem 5.1: Relation between target and actual returns
• Remark 5.1: Smoothness
• Theorem 6.1: Safety guarantee
• Theorem 6.2: Near-optimality
• Remark C.1
• Remark C.2
• Remark C.3
• Theorem D.1
• Remark D.1
• Theorem D.2