Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Novel Multi-Timescale Stability-Preserving Hierarchical Reinforcement Learning Controller Framework for Adaptive Control in High-Dimensional Dynamical Systems

Mohammad Ali Labbaf Khaniki, Fateme Taroodi, Benyamin Safizadeh

TL;DR

The paper tackles robust control of high-dimensional stochastic systems by combining hierarchical decision-making with formal stochastic stability. It introduces Multi-Timescale Lyapunov-Constrained Hierarchical Reinforcement Learning (MTLHRL), an SMDP-based framework that couples a high-level planner with a low-level controller while enforcing a neural Lyapunov constraint via Lagrangian relaxation and multi-timescale updates. Theoretical results guarantee mean-square boundedness or asymptotic stability, and practical implementations include neural Lyapunov representations and trust-region updates. Empirical validation on an 8D hyperchaotic system and a 5-DOF robotic manipulator shows improved stability, faster convergence, and better disturbance rejection compared with PPO, DDPG, and STLHRL. Overall, MTLHRL offers a scalable, stability-certified approach for safe, data-efficient control of complex stochastic dynamics with potential impact in robotics, autonomous systems, and chaotic synchronization.

Abstract

Controlling high-dimensional stochastic systems, critical in robotics, autonomous vehicles, and hyperchaotic systems, faces the curse of dimensionality, lacks temporal abstraction, and often fails to ensure stochastic stability. To overcome these limitations, this study introduces the Multi-Timescale Lyapunov-Constrained Hierarchical Reinforcement Learning (MTLHRL) framework. MTLHRL integrates a hierarchical policy within a semi-Markov Decision Process (SMDP), featuring a high-level policy for strategic planning and a low-level policy for reactive control, which effectively manages complex, multi-timescale decision-making and reduces dimensionality overhead. Stability is rigorously enforced using a neural Lyapunov function optimized via Lagrangian relaxation and multi-timescale actor-critic updates, ensuring mean-square boundedness or asymptotic stability in the face of stochastic dynamics. The framework promotes efficient and reliable learning through trust-region constraints and decoupled optimization. Extensive simulations on an 8D hyperchaotic system and a 5-DOF robotic manipulator demonstrate MTLHRL's empirical superiority. It significantly outperforms baseline methods in both stability and performance, recording the lowest error indices (e.g., Integral Absolute Error (IAE): 3.912 in hyperchaotic control and IAE: 1.623 in robotics), achieving faster convergence, and exhibiting superior disturbance rejection. MTLHRL offers a theoretically grounded and practically viable solution for robust control of complex stochastic systems.

A Novel Multi-Timescale Stability-Preserving Hierarchical Reinforcement Learning Controller Framework for Adaptive Control in High-Dimensional Dynamical Systems

TL;DR

The paper tackles robust control of high-dimensional stochastic systems by combining hierarchical decision-making with formal stochastic stability. It introduces Multi-Timescale Lyapunov-Constrained Hierarchical Reinforcement Learning (MTLHRL), an SMDP-based framework that couples a high-level planner with a low-level controller while enforcing a neural Lyapunov constraint via Lagrangian relaxation and multi-timescale updates. Theoretical results guarantee mean-square boundedness or asymptotic stability, and practical implementations include neural Lyapunov representations and trust-region updates. Empirical validation on an 8D hyperchaotic system and a 5-DOF robotic manipulator shows improved stability, faster convergence, and better disturbance rejection compared with PPO, DDPG, and STLHRL. Overall, MTLHRL offers a scalable, stability-certified approach for safe, data-efficient control of complex stochastic dynamics with potential impact in robotics, autonomous systems, and chaotic synchronization.

Abstract

Controlling high-dimensional stochastic systems, critical in robotics, autonomous vehicles, and hyperchaotic systems, faces the curse of dimensionality, lacks temporal abstraction, and often fails to ensure stochastic stability. To overcome these limitations, this study introduces the Multi-Timescale Lyapunov-Constrained Hierarchical Reinforcement Learning (MTLHRL) framework. MTLHRL integrates a hierarchical policy within a semi-Markov Decision Process (SMDP), featuring a high-level policy for strategic planning and a low-level policy for reactive control, which effectively manages complex, multi-timescale decision-making and reduces dimensionality overhead. Stability is rigorously enforced using a neural Lyapunov function optimized via Lagrangian relaxation and multi-timescale actor-critic updates, ensuring mean-square boundedness or asymptotic stability in the face of stochastic dynamics. The framework promotes efficient and reliable learning through trust-region constraints and decoupled optimization. Extensive simulations on an 8D hyperchaotic system and a 5-DOF robotic manipulator demonstrate MTLHRL's empirical superiority. It significantly outperforms baseline methods in both stability and performance, recording the lowest error indices (e.g., Integral Absolute Error (IAE): 3.912 in hyperchaotic control and IAE: 1.623 in robotics), achieving faster convergence, and exhibiting superior disturbance rejection. MTLHRL offers a theoretically grounded and practically viable solution for robust control of complex stochastic systems.
Paper Structure (19 sections, 4 theorems, 36 equations, 6 figures, 2 tables, 1 algorithm)

This paper contains 19 sections, 4 theorems, 36 equations, 6 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundational Work in RL and Control for Stochastic Systems
  4. Recent Advances in Hierarchical RL and Stability-Constrained Methods
  5. Specific Themes: Multi-Timescale Optimization and Applications
  6. Gaps and Motivation for the Proposed Study
  7. Methodology
  8. Overview
  9. Multi-Timescale Lyapunov-Constrained Hierarchical Reinforcement Learning Framework
  10. Stability Analysis and Optimization Convergence
  11. Implementation and Algorithmic Details
  12. Neural Network Architectures
  13. Training Procedure
  14. Algorithmic Optimizations
  15. Simulation Results
  16. ...and 4 more sections

Key Result

Lemma 3.1

Consider the system governed by eq:sde, with policy $\pi(x; \theta)$ parameterized by a neural network possessing universal approximation capability, and a Lyapunov function $V(x; \phi): \mathbb{R}^n \to \mathbb{R}_{\geq 0}$ satisfying Definition def:lyapunov_function. Assume the drift $f(x, u)$ is Moreover, if controllability allows sufficient negative drift (e.g., $M = -\infty$ for unbounded ac

Figures (6)

  • Figure 1: Block Diagram of Multi-Timescale Lyapunov-Constrained Hierarchical Reinforcement Learning (MTLHRL) framework.
  • Figure 2: Learning Curves Comparison: MTLHRL vs Baselines.
  • Figure 3: Performance comparison among four approaches—PPO, DDPG, STLHRL, MTLHRL—for all eight states.
  • Figure 4: Euclidean norms of synchronization errors under PPO, DDPG, STLHRL, and MTLHRL for the eight states.
  • Figure 5: Euclidean norm of state errors for four controllers—PPO, DDPG, STLHRL, MTLHRL—across states of the 5-DOF robot manipulator system.
  • ...and 1 more figures

Theorems & Definitions (14)

  • Definition 3.1: Mean-Square Boundedness
  • Definition 3.2: Asymptotic Mean-Square Stability
  • Definition 3.3: Neural Lyapunov Function
  • Lemma 3.1: Feasibility of Stochastic Stability Constraint
  • proof
  • Remark 3.1
  • Lemma 3.2: Boundedness of Lyapunov Loss
  • proof
  • Lemma 3.3: Convergence of Multi-Timescale Updates
  • proof
  • ...and 4 more