Safe Model-based Reinforcement Learning with Stability Guarantees

TL;DR

The paper tackles the challenge of safety in reinforcement learning for safety-critical systems by marrying Lyapunov stability certificates with Gaussian-process-based dynamic models. It introduces SafeLyapunovLearning, a model-based RL framework that starts from a known safe policy, constructs high-probability regions of attraction, and safely expands this region through data collection within the safe set while improving control performance. Theoretical guarantees show how to compute and enlarge the ROA under Lipschitz assumptions and GP confidence bounds, and a practical algorithm demonstrates safe optimization of a neural policy for inverted pendulum stabilization. The work provides both rigorous safety guarantees and a workable, data-efficient learning approach that could enable RL in real-world, safety-critical applications.

Abstract

Reinforcement learning is a powerful paradigm for learning optimal policies from experimental data. However, to find optimal policies, most reinforcement learning algorithms explore all possible actions, which may be harmful for real-world systems. As a consequence, learning algorithms are rarely applied on safety-critical systems in the real world. In this paper, we present a learning algorithm that explicitly considers safety, defined in terms of stability guarantees. Specifically, we extend control-theoretic results on Lyapunov stability verification and show how to use statistical models of the dynamics to obtain high-performance control policies with provable stability certificates. Moreover, under additional regularity assumptions in terms of a Gaussian process prior, we prove that one can effectively and safely collect data in order to learn about the dynamics and thus both improve control performance and expand the safe region of the state space. In our experiments, we show how the resulting algorithm can safely optimize a neural network policy on a simulated inverted pendulum, without the pendulum ever falling down.

Figures (2)

• Figure 1: Example application of \ref{['alg:safe_learning']}. Due to input constraints, the system becomes unstable for large states. We start from an initial, local policy $\pi_0$ that has a small, safe region of attraction (red lines) in \ref{['fig:example_set_1']}. The algorithm selects safe, informative state-action pairs within $\mathcal{S}_n$ (top, white shaded), which can be evaluated without leaving the region of attraction $\mathcal{V}(c_n)$ (red lines) of the current policy $\pi_n$. As we gather more data (blue crosses), the uncertainty in the model decreases (top, background) and we use \ref{['eq:policy_update']} to update the policy so that it lies within $\mathcal{D}_n$ (top, red shaded) and fulfills the Lyapunov decrease condition. The algorithm converges to the largest safe set in \ref{['fig:example_set_3']}. It improves the policy without evaluating unsafe state-action pairs and thereby without system failure.
• Figure 2: Optimization results for an inverted pendulum. \ref{['fig:levelset']} shows the initial safe set (yellow) under the policy $\pi_0$, while the green region represents the estimated region of attraction under the optimized neural network policy. It is contained within the true region of attraction (white). \ref{['fig:performance']} shows the improved performance of the safely learned policy over the policy for the prior model.

Theorems & Definitions (46)

• Theorem 1: Khalil1996Nonlinear
• Theorem 2
• Theorem 3
• Theorem 4
• Lemma 1
• proof
• Lemma 2
• proof
• Corollary 1
• proof