Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MSACL: Multi-Step Actor-Critic Learning with Lyapunov Certificates for Exponentially Stabilizing Control

Yongwei Zhang, Yuanzhe Xing, Quan Quan, Zhikun She

TL;DR

MSACL tackles the challenge of provable stability in model-free reinforcement learning for continuous-control tasks by integrating exponential stability theory with maximum entropy RL through multi-step Lyapunov certificate learning. It introduces Exponential Stability Labels (ESL), a $λ$-weighted aggregation, and an off-policy multi-step data framework to learn Lyapunov certificates and stability-aware policies. The method achieves exponential stability and rapid convergence across six nonlinear benchmarks (stabilization and tracking) with robustness to disturbances and unseen trajectories, identifying $n=20$ as a robust default. By bridging Lyapunov theory and off-policy actor-critic methods, MSACL provides verifiably safe learning-based control and offers a principled path toward deployment in safety-critical systems.

Abstract

Achieving provable stability in model-free reinforcement learning (RL) remains a challenge, particularly in balancing exploration with rigorous safety. This article introduces MSACL, a framework that integrates exponential stability theory with maximum entropy RL through multi-step Lyapunov certificate learning. Unlike methods relying on complex reward engineering, MSACL utilizes off-policy multi-step data to learn Lyapunov certificates satisfying theoretical stability conditions. By introducing Exponential Stability Labels (ESL) and a $λ$-weighted aggregation mechanism, the framework effectively balances the bias-variance trade-off in multi-step learning. Policy optimization is guided by a stability-aware advantage function, ensuring the learned policy promotes rapid Lyapunov descent. We evaluate MSACL across six benchmarks, including stabilization and nonlinear tracking tasks, demonstrating its superiority over state-of-the-art Lyapunov-based RL algorithms. MSACL achieves exponential stability and rapid convergence under simple rewards, while exhibiting significant robustness to uncertainties and generalization to unseen trajectories. Sensitivity analysis establishes the multi-step horizon $n=20$ as a robust default across diverse systems. By linking Lyapunov theory with off-policy actor-critic frameworks, MSACL provides a foundation for verifiably safe learning-based control. Source code and benchmark environments will be made publicly available.

MSACL: Multi-Step Actor-Critic Learning with Lyapunov Certificates for Exponentially Stabilizing Control

TL;DR

MSACL tackles the challenge of provable stability in model-free reinforcement learning for continuous-control tasks by integrating exponential stability theory with maximum entropy RL through multi-step Lyapunov certificate learning. It introduces Exponential Stability Labels (ESL), a -weighted aggregation, and an off-policy multi-step data framework to learn Lyapunov certificates and stability-aware policies. The method achieves exponential stability and rapid convergence across six nonlinear benchmarks (stabilization and tracking) with robustness to disturbances and unseen trajectories, identifying as a robust default. By bridging Lyapunov theory and off-policy actor-critic methods, MSACL provides verifiably safe learning-based control and offers a principled path toward deployment in safety-critical systems.

Abstract

Achieving provable stability in model-free reinforcement learning (RL) remains a challenge, particularly in balancing exploration with rigorous safety. This article introduces MSACL, a framework that integrates exponential stability theory with maximum entropy RL through multi-step Lyapunov certificate learning. Unlike methods relying on complex reward engineering, MSACL utilizes off-policy multi-step data to learn Lyapunov certificates satisfying theoretical stability conditions. By introducing Exponential Stability Labels (ESL) and a -weighted aggregation mechanism, the framework effectively balances the bias-variance trade-off in multi-step learning. Policy optimization is guided by a stability-aware advantage function, ensuring the learned policy promotes rapid Lyapunov descent. We evaluate MSACL across six benchmarks, including stabilization and nonlinear tracking tasks, demonstrating its superiority over state-of-the-art Lyapunov-based RL algorithms. MSACL achieves exponential stability and rapid convergence under simple rewards, while exhibiting significant robustness to uncertainties and generalization to unseen trajectories. Sensitivity analysis establishes the multi-step horizon as a robust default across diverse systems. By linking Lyapunov theory with off-policy actor-critic frameworks, MSACL provides a foundation for verifiably safe learning-based control. Source code and benchmark environments will be made publicly available.
Paper Structure (52 sections, 1 theorem, 48 equations, 8 figures, 17 tables, 1 algorithm)

This paper contains 52 sections, 1 theorem, 48 equations, 8 figures, 17 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Statement
  4. Exponential Stability and Lyapunov Certificates
  5. Standard RL and Maximum Entropy RL
  6. Learning Lyapunov Certificate
  7. Data Collection
  8. Loss Function Design of Lyapunov Network
  9. Learning Policy with Exponential Stability Constraints
  10. Loss Function Design of Soft Q Networks
  11. Loss Function Design of Policy Network
  12. Automated Temperature Adjustment
  13. Experiments
  14. Benchmark Environments
  15. Baseline Algorithms
  16. ...and 37 more sections

Key Result

Lemma 1

For the control system (control_system), if there exists a continuous function $V: \mathcal{X} \mapsto \mathbb{R}$ and positive constants $\alpha_1, \alpha_2 > 0$ and $0 < \alpha_3 < 1$ satisfying: for all $t \in \mathbb{Z}_{\geq 0}$, where $\mathbf{x}_{t+1} = f(\mathbf{x}_t, \mathbf{u}_t)$, then the origin $\mathbf{x}_g = \mathbf{0}$ is exponentially stable. The function $V$ is referred to as an

Figures (8)

  • Figure 1: Schematic of the $n$-step sliding window data collection process. A double-ended queue (deque) of size $n$ is utilized to buffer consecutive interaction transitions $\mathbf{d}_t$, ensuring that complete $n$-step trajectories are available for the verification of the multi-step Lyapunov certificate.
  • Figure 2: Flowchart of the proposed MSACL algorithm.
  • Figure 3: Benchmark environments. (a) Uncontrolled VanderPol (Phase Portrait), (b) Pendulum, (c) Planar DuctedFan, (d) Two-link planar robot, (e) SingleCarTracking, (f) QuadrotorTracking.
  • Figure 4: Training performance across six benchmark environments. The results show the average MCR and MCC over five independent runs. Solid lines and shaded regions represent the mean and one standard deviation, respectively. The horizontal axis indicates the number of training steps.
  • Figure 5: Visualization of learned Lyapunov certificates and their corresponding contour plots across various benchmark environments: (a) VanderPol; (b) Pendulum; (c)--(e) DuctedFan (projections in the $(x,y)$-plane with $\theta=0$, $(y,\theta)$-plane with $x=0$, and $(x,\theta)$-plane with $y=0$); (f) Two-link; (g) SingleCarTracking; (h)--(j) QuadrotorTracking (projections in the $(e_x,e_y)$-plane with $e_z=0$, $(e_y,e_z)$-plane with $e_x=0$, and $(e_x,e_z)$-plane with $e_y=0$).
  • ...and 3 more figures

Theorems & Definitions (7)

  • Definition 1
  • Lemma 1
  • Definition 2
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4