Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions
Kehan Long, Jorge Cortés, Nikolay Atanasov
TL;DR
The paper introduces generalized Lyapunov functions that augment RL value functions with neural residuals and replace the classic one-step decrease with a multi-step, weighted decrease to certify stability of learned policies. It analyzes linear-quadratic settings via LMIs and extends the framework to nonlinear RL by jointly learning policies and stability certificates, validated on standard benchmarks. The approach yields larger regions of attraction and enables stability certificates for nonlinear RL, while also enabling joint policy-certificate synthesis, albeit with increased verification complexity. This work bridges classical control theory and modern learning, offering a scalable pathway toward formal stability guarantees for learned policies and their deployment.
Abstract
Establishing stability certificates for closed-loop systems under reinforcement learning (RL) policies is essential to move beyond empirical performance and offer guarantees of system behavior. Classical Lyapunov methods require a strict stepwise decrease in the Lyapunov function but such certificates are difficult to construct for learned policies. The RL value function is a natural candidate but it is not well understood how it can be adapted for this purpose. To gain intuition, we first study the linear quadratic regulator (LQR) problem and make two key observations. First, a Lyapunov function can be obtained from the value function of an LQR policy by augmenting it with a residual term related to the system dynamics and stage cost. Second, the classical Lyapunov decrease requirement can be relaxed to a generalized Lyapunov condition requiring only decrease on average over multiple time steps. Using this intuition, we consider the nonlinear setting and formulate an approach to learn generalized Lyapunov functions by augmenting RL value functions with neural network residual terms. Our approach successfully certifies the stability of RL policies trained on Gymnasium and DeepMind Control benchmarks. We also extend our method to jointly train neural controllers and stability certificates using a multi-step Lyapunov loss, resulting in larger certified inner approximations of the region of attraction compared to the classical Lyapunov approach. Overall, our formulation enables stability certification for a broad class of systems with learned policies by making certificates easier to construct, thereby bridging classical control theory and modern learning-based methods.