Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lyapunov Function-guided Reinforcement Learning for Flight Control

Yifei Li, Erik-Jan van Kampen

TL;DR

This work introduces a Lyapunov-function-guided approach to reinforcement learning for cascaded flight control, explicitly shaping the learning objective to promote convergence of tracking errors in a nonlinear, time-varying system. By deriving a discrete-time Lyapunov increment that accounts for discretization and model-approximation errors, the authors integrate a convergence metric into the IHDP policy optimization, balancing tracking performance and action smoothness. Simulations show modest improvements in convergence for the higher-level controller and clearer gains in tracking smoothness for the lower-level controller when the convergence loss is activated, highlighting trade-offs between aggressive convergence and input smoothness. The proposed method provides a practical framework for incorporating stability-aware metrics into data-driven flight control, with potential impact on safer and smoother autonomous flight operations.

Abstract

A cascaded online learning flight control system has been developed and enhanced with respect to action smoothness. In this paper, we investigate the convergence performance of the control system, characterized by the increment of a Lyapunov function candidate. The derivation of this metric accounts for discretization errors and state prediction errors introduced by the incremental model. Comparative results are presented through flight control simulations.

Lyapunov Function-guided Reinforcement Learning for Flight Control

TL;DR

This work introduces a Lyapunov-function-guided approach to reinforcement learning for cascaded flight control, explicitly shaping the learning objective to promote convergence of tracking errors in a nonlinear, time-varying system. By deriving a discrete-time Lyapunov increment that accounts for discretization and model-approximation errors, the authors integrate a convergence metric into the IHDP policy optimization, balancing tracking performance and action smoothness. Simulations show modest improvements in convergence for the higher-level controller and clearer gains in tracking smoothness for the lower-level controller when the convergence loss is activated, highlighting trade-offs between aggressive convergence and input smoothness. The proposed method provides a practical framework for incorporating stability-aware metrics into data-driven flight control, with potential impact on safer and smoother autonomous flight operations.

Abstract

A cascaded online learning flight control system has been developed and enhanced with respect to action smoothness. In this paper, we investigate the convergence performance of the control system, characterized by the increment of a Lyapunov function candidate. The derivation of this metric accounts for discretization errors and state prediction errors introduced by the incremental model. Comparative results are presented through flight control simulations.
Paper Structure (15 sections, 28 equations, 5 figures, 2 tables)

This paper contains 15 sections, 28 equations, 5 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. Model
  3. Dynamics
  4. Tracking dynamics
  5. Equilibrium Point Analysis
  6. Explaination of $e_{1}$ divergence
  7. Lyapunov Function-guided IHDP
  8. Discrete-time Lyapunov function increment
  9. Modification on IHDP
  10. Simulation
  11. Reward shaping
  12. Results and discussion
  13. Higher-level agent
  14. Lower-level agent
  15. Conclusion

Figures (5)

  • Figure 1: Comparison of angle-of-attack tracking between IHDP and Lyapunov function-guided IHDP used by the higher-level agent.
  • Figure 2: Comparison of critic output and its increment for the higher-level agent.
  • Figure 3: Comparison of pitch rate tracking between IHDP and Lyapunov function-guided IHDP used by the lower-level agent.
  • Figure 4: Comparison of pitch rate tracking error and control surface deflection between IHDP and Lyapunov function-guided IHDP for the lower-level agent.
  • Figure 5: Comparison of critic output and its increment for the lower-level agent.