Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stability Enhancement in Reinforcement Learning via Adaptive Control Lyapunov Function

Donghe Chen, Han Wang, Lin Cheng, Shengping Gong

TL;DR

The paper tackles the safety gap in reinforcement learning for real-world control by introducing SAC-CLF, which fuses Soft Actor-Critic with Control Lyapunov Functions. It advances safety during learning through (i) a systematic, task-specific CLF design via system linearization and LQR, (ii) adaptive constraint strength informed by CLF derivative discrepancies, and (iii) a vibration-dampening term that prioritizes smooth control inputs. Across simulations on a nonlinear 2D system and spacecraft attitude control, SAC-CLF improves stability, robustness to unmodeled dynamics, and sample efficiency compared to baselines, with notable gains for the satellite scenario. The work supports safer, more reliable RL-based control in safety-critical applications and points to extensions to more complex dynamics and autonomous systems.

Abstract

Reinforcement Learning (RL) has shown promise in control tasks but faces significant challenges in real-world applications, primarily due to the absence of safety guarantees during the learning process. Existing methods often struggle with ensuring safe exploration, leading to potential system failures and restricting applications primarily to simulated environments. Traditional approaches such as reward shaping and constrained policy optimization can fail to guarantee safety during initial learning stages, while model-based methods using Control Lyapunov Functions (CLFs) or Control Barrier Functions (CBFs) may hinder efficient exploration and performance. To address these limitations, this paper introduces Soft Actor-Critic with Control Lyapunov Function (SAC-CLF), a framework that enhances stability and safety through three key innovations: (1) a task-specific CLF design method for safe and optimal performance; (2) dynamic adjustment of constraints to maintain robustness under unmodeled dynamics; and (3) improved control input smoothness while ensuring safety. Experimental results on a classical nonlinear system and satellite attitude control demonstrate the effectiveness of SAC-CLF in overcoming the shortcomings of existing methods.

Stability Enhancement in Reinforcement Learning via Adaptive Control Lyapunov Function

TL;DR

The paper tackles the safety gap in reinforcement learning for real-world control by introducing SAC-CLF, which fuses Soft Actor-Critic with Control Lyapunov Functions. It advances safety during learning through (i) a systematic, task-specific CLF design via system linearization and LQR, (ii) adaptive constraint strength informed by CLF derivative discrepancies, and (iii) a vibration-dampening term that prioritizes smooth control inputs. Across simulations on a nonlinear 2D system and spacecraft attitude control, SAC-CLF improves stability, robustness to unmodeled dynamics, and sample efficiency compared to baselines, with notable gains for the satellite scenario. The work supports safer, more reliable RL-based control in safety-critical applications and points to extensions to more complex dynamics and autonomous systems.

Abstract

Reinforcement Learning (RL) has shown promise in control tasks but faces significant challenges in real-world applications, primarily due to the absence of safety guarantees during the learning process. Existing methods often struggle with ensuring safe exploration, leading to potential system failures and restricting applications primarily to simulated environments. Traditional approaches such as reward shaping and constrained policy optimization can fail to guarantee safety during initial learning stages, while model-based methods using Control Lyapunov Functions (CLFs) or Control Barrier Functions (CBFs) may hinder efficient exploration and performance. To address these limitations, this paper introduces Soft Actor-Critic with Control Lyapunov Function (SAC-CLF), a framework that enhances stability and safety through three key innovations: (1) a task-specific CLF design method for safe and optimal performance; (2) dynamic adjustment of constraints to maintain robustness under unmodeled dynamics; and (3) improved control input smoothness while ensuring safety. Experimental results on a classical nonlinear system and satellite attitude control demonstrate the effectiveness of SAC-CLF in overcoming the shortcomings of existing methods.
Paper Structure (16 sections, 2 theorems, 33 equations, 8 figures, 3 tables, 1 algorithm)

This paper contains 16 sections, 2 theorems, 33 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Statement
  4. Soft Actor-Critic
  5. Integrating Control Lyapunov Functions with Quadratic Programming
  6. CLF-Based Compensating Control with Reinforcement Learning
  7. Task-Specific CLFs Design
  8. Adaptive Constraint Strength
  9. Safety-Prioritized Control Input Smoothing
  10. Summary
  11. Simulation
  12. Environments
  13. Validation of task-specific CLF Design
  14. Validation of Adaptive Constraints Strength
  15. Validation of Safety-Prioritized Control Input Smoothing
  16. ...and 1 more sections

Key Result

Theorem 1

A function $V: \mathbb{R}^n \to \mathbb{R}$ is termed a control Lyapunov function for a system if it satisfies: These conditions ensure $V$ can be used to design stabilizing feedback controls, rendering the closed-loop system asymptotically stable at the origin. The CLF concept is not limited to quadratic forms but applies to any function meeting these criteria.

Figures (8)

  • Figure 1: Framework of SAC-CLF
  • Figure 2: The hierarchical relationships: the state space contains all possible states; within it, the safe state space $\mathcal{B}$ includes states that ensure safety; and further within $\mathcal{B}$, the safe energy ball $\mathcal{D}$ comprises states with a control Lyapunov function below a threshold $V_0$.
  • Figure 3: The CLF from an LQR-based control policy ensures both stability and local optimality, with the CLF and value function closely matching near the equilibrium.
  • Figure 4: Optimal control input may not satisfy CLF constraints, compromising optimality.
  • Figure 5: Safety-Prioritized Control Input Smoothing: Comparative Analysis Between the Proposed Method and Naive Smoothing Techniques
  • ...and 3 more figures

Theorems & Definitions (5)

  • Definition 1: Nominal System
  • Theorem 1: Control Lyapunov Function
  • Definition 2: Safe State Space
  • Definition 3: Safe Energy Ball
  • Theorem 2