Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lyapunov Constrained Soft Actor-Critic (LC-SAC) using Koopman Operator Theory for Quadrotor Trajectory Tracking

Dhruv S. Kushwaha, Zoleikha A. Biron

TL;DR

The paper tackles stability and safety in reinforcement learning for safety-critical systems by integrating Lyapunov stability guarantees into Soft Actor-Critic (SAC) via a Lyapunov-constrained framework. It leverages Koopman operator theory and EDMD to obtain a linear lifted surrogate of the nonlinear dynamics and derives a closed-form discrete-time control Lyapunov function (CLF) by solving the discrete algebraic Riccati equation, enabling an efficient Lyapunov decrease check. The online policy optimization adds a Lagrangian-penalized Lyapunov term to the SAC objective, with a dual update that enforces the decrease constraint on average, yielding exponential convergence properties in the lifted space under the surrogate model. Experiments on a 2D quadrotor trajectory-tracking task show LC-SAC enhances stability, reduces worst-case Lyapunov violations, and delivers more reliable evaluation performance than vanilla SAC, demonstrating practical benefits for safety-aware RL in robotics. The approach provides a principled, data-driven route to stability guarantees without training an explicit Lyapunov network, while highlighting opportunities to address model mismatch and integrate additional safety constraints.

Abstract

Reinforcement Learning (RL) has achieved remarkable success in solving complex sequential decision-making problems. However, its application to safety-critical physical systems remains constrained by the lack of stability guarantees. Standard RL algorithms prioritize reward maximization, often yielding policies that may induce oscillations or unbounded state divergence. There has significant work in incorporating Lyapunov-based stability guarantees in RL algorithms with key challenges being selecting a candidate Lyapunov function, computational complexity by using excessive function approximators and conservative policies by incorporating stability criterion in the learning process. In this work we propose a novel Lyapunov-constrained Soft Actor-Critic (LC-SAC) algorithm using Koopman operator theory. We propose use of extended dynamic mode decomposition (EDMD) to produce a linear approximation of the system and use this approximation to derive a closed form solution for candidate Lyapunov function. This derived Lyapunov function is incorporated in the SAC algorithm to further provide guarantees for a policy that stabilizes the nonlinear system. The results are evaluated trajectory tracking of a 2D Quadrotor environment based on safe-control-gym. The proposed algorithm shows training convergence and decaying violations for Lyapunov stability criterion compared to baseline vanilla SAC algorithm. GitHub Repository: https://github.com/DhruvKushwaha/LC-SAC-Quadrotor-Trajectory-Tracking

Lyapunov Constrained Soft Actor-Critic (LC-SAC) using Koopman Operator Theory for Quadrotor Trajectory Tracking

TL;DR

The paper tackles stability and safety in reinforcement learning for safety-critical systems by integrating Lyapunov stability guarantees into Soft Actor-Critic (SAC) via a Lyapunov-constrained framework. It leverages Koopman operator theory and EDMD to obtain a linear lifted surrogate of the nonlinear dynamics and derives a closed-form discrete-time control Lyapunov function (CLF) by solving the discrete algebraic Riccati equation, enabling an efficient Lyapunov decrease check. The online policy optimization adds a Lagrangian-penalized Lyapunov term to the SAC objective, with a dual update that enforces the decrease constraint on average, yielding exponential convergence properties in the lifted space under the surrogate model. Experiments on a 2D quadrotor trajectory-tracking task show LC-SAC enhances stability, reduces worst-case Lyapunov violations, and delivers more reliable evaluation performance than vanilla SAC, demonstrating practical benefits for safety-aware RL in robotics. The approach provides a principled, data-driven route to stability guarantees without training an explicit Lyapunov network, while highlighting opportunities to address model mismatch and integrate additional safety constraints.

Abstract

Reinforcement Learning (RL) has achieved remarkable success in solving complex sequential decision-making problems. However, its application to safety-critical physical systems remains constrained by the lack of stability guarantees. Standard RL algorithms prioritize reward maximization, often yielding policies that may induce oscillations or unbounded state divergence. There has significant work in incorporating Lyapunov-based stability guarantees in RL algorithms with key challenges being selecting a candidate Lyapunov function, computational complexity by using excessive function approximators and conservative policies by incorporating stability criterion in the learning process. In this work we propose a novel Lyapunov-constrained Soft Actor-Critic (LC-SAC) algorithm using Koopman operator theory. We propose use of extended dynamic mode decomposition (EDMD) to produce a linear approximation of the system and use this approximation to derive a closed form solution for candidate Lyapunov function. This derived Lyapunov function is incorporated in the SAC algorithm to further provide guarantees for a policy that stabilizes the nonlinear system. The results are evaluated trajectory tracking of a 2D Quadrotor environment based on safe-control-gym. The proposed algorithm shows training convergence and decaying violations for Lyapunov stability criterion compared to baseline vanilla SAC algorithm. GitHub Repository: https://github.com/DhruvKushwaha/LC-SAC-Quadrotor-Trajectory-Tracking
Paper Structure (21 sections, 2 theorems, 66 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 21 sections, 2 theorems, 66 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Theoretical Background
  3. Lyapunov Functions (Discrete-Time)
  4. Discrete-Time Control Lyapunov Functions.
  5. Soft Actor-Critic Algorithm
  6. Koopman Operator Theory
  7. Proposed Methodology
  8. Critic update (standard SAC).
  9. Actor update (Lyapunov constrained, Lagrangian form).
  10. Stability Analysis
  11. Surrogate dynamics and CLF
  12. Constraint enforced by LC-SAC
  13. From \ref{['eqn:dec_a_norm']} to exponential stability (lifted system).
  14. How the LC-SAC updates reduce violation
  15. Dual update enforces constraint on average
  16. ...and 6 more sections

Key Result

Theorem 1

khalil2002nonlinear Consider a discrete-time closed-loop system with desired (equilibrium) state $x_d \in \mathcal{X}$. A continuously differentiable function $V:\mathcal{X}\to \mathbb{R}$ is a (discrete-time) Lyapunov function if:

Figures (7)

  • Figure 1: Koopman Operator: State trajectories $x_t$ and observable trajectories $y_t:=g(x_t)$.
  • Figure 2: Proposed methodology for Lyapunov-based SAC.
  • Figure 3: Lyapunov Loss decay for LC-SAC over 5 trials.
  • Figure 4: Average evaluation rewards during training for proposed LC-SAC and baseline SAC.
  • Figure 5: Mean and Variance trajectory comparison for LC-SAC vs baseline SAC in X-Z plane.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2