Off-Policy Reinforcement Learning with Anytime Safety Guarantees via Robust Safe Gradient Flow

TL;DR

This work tackles constrained RL with anytime safety, ensuring constraint satisfaction at every iteration. It introduces RSGF-RL, an off-policy method built on the Robust Safe Gradient Flow that updates policies via a convex QCQP using off-policy estimates of $V_j(\theta)$ and $\nabla V_j(\theta)$, and enforces forward invariance of the safe set. The authors establish well-posedness of RSGF, equivalence of equilibria with $KKT$ points, and almost-sure convergence to the $KKT$ set, together with finite-iteration bounds and statistical concentration results for the estimators. They demonstrate the approach on navigation and cart-pole benchmarks, showing safety guarantees and superior performance relative to state-of-the-art safe RL methods, including efficient use of off-policy data. The work advances safe RL by enabling off-policy data usage, rigorous probabilistic safety guarantees, and solid convergence theory grounded in stochastic approximation.

Abstract

This paper considers the problem of solving constrained reinforcement learning (RL) problems with anytime guarantees, meaning that the algorithmic solution must yield a constraint-satisfying policy at every iteration of its evolution. Our design is based on a discretization of the Robust Safe Gradient Flow (RSGF), a continuous-time dynamics for anytime constrained optimization whose forward invariance and stability properties we formally characterize. The proposed strategy, termed RSGF-RL, is an off-policy algorithm which uses episodic data to estimate the value functions and their gradients and updates the policy parameters by solving a convex quadratically constrained quadratic program. Our technical analysis combines statistical analysis, the theory of stochastic approximation, and convex analysis to determine the number of episodes sufficient to ensure that safe policies are updated to safe policies and to recover from an unsafe policy, both with an arbitrary user-specified probability, and to establish the asymptotic convergence to the set of KKT points of the RL problem almost surely. Simulations on a navigation example and the cart-pole system illustrate the superior performance of RSGF-RL with respect to the state of the art.

Figures (3)

• Figure 1: Policy evolution under RSGF-RL in the Navigation 2D example. Obstacles are depicted in gray. Target point in green and different robot initial conditions in light blue. Initial policy on the left, final policy on the right, with intermediate policies obtained during the algorithm evolution in the middle.
• Figure 2: Comparison between CPO and different RSGF-RL training strategies in the Navigation 2D environment. Left plot shows the average $V_0( \theta)$ as a performance metric, while the right plot shows the average $V_1(\theta)$ as a safety metric. Averages are computed over 5 seeds and the shaded area represents the standard deviation.
• Figure 3: Comparison between CPO and different RSGF-RL training strategies in the Cart-pole environment. Left plot shows the average $V_0( \theta)$ as a performance metric, while the right plot shows the average $V_1(\theta)$ as a safety metric. The max reward is 126.61. Averages are computed over 5 seeds and the shaded area represents the standard deviation.

Theorems & Definitions (38)

• Remark 3.1
• Lemma 4.1
• proof
• Lemma 4.2
• proof
• Proposition 4.3
• proof
• Proposition 4.4
• proof
• Proposition 4.5