Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep neural network approximations for the stable manifolds of the Hamilton-Jacobi-Bellman equations

Guoyuan Chen

TL;DR

The paper tackles infinite-horizon optimal control by representing the optimal feedback through the stable manifold of the Hamiltonian system associated with the stationary HJB equation. It proves that, under suitable accuracy, a neural network approximation of the stable manifold yields a stabilizing, nearly optimal closed-loop and provides exponential convergence bounds. Building on this theory, the authors develop a grid-free, data-driven algorithm that uses adaptive trajectory generation via two-point boundary-value problems and forward integration, a composite loss, and rescaling to train an LSTM-type network that approximates the manifold. They validate the approach on two high-dimensional problems—the Reaction Wheel Pendulum and the parabolic Allen-Cahn equation—demonstrating real-time control performance and substantial improvements over traditional methods, with the method remaining scalable to high-dimensional systems.

Abstract

For an infinite-horizon control problem, the optimal control can be represented by the stable manifold of the characteristic Hamiltonian system of Hamilton-Jacobi-Bellman (HJB) equation in a semiglobal domain. In this paper, we first theoretically prove that if an approximation is sufficiently close to the exact stable manifold of the HJB equation in a certain sense, then the control derived from this approximation stabilizes the system and is nearly optimal. Then, based on the theoretical result, we propose a deep learning algorithm to approximate the stable manifold and compute optimal feedback control numerically. The algorithm relies on adaptive data generation through finding trajectories randomly within the stable manifold. Such kind of algorithm is grid-free basically, making it potentially applicable to a wide range of high-dimensional nonlinear systems. We demonstrate the effectiveness of our method through two examples: stabilizing the Reaction Wheel Pendulums and controlling the parabolic Allen-Cahn equation.

Deep neural network approximations for the stable manifolds of the Hamilton-Jacobi-Bellman equations

TL;DR

The paper tackles infinite-horizon optimal control by representing the optimal feedback through the stable manifold of the Hamiltonian system associated with the stationary HJB equation. It proves that, under suitable accuracy, a neural network approximation of the stable manifold yields a stabilizing, nearly optimal closed-loop and provides exponential convergence bounds. Building on this theory, the authors develop a grid-free, data-driven algorithm that uses adaptive trajectory generation via two-point boundary-value problems and forward integration, a composite loss, and rescaling to train an LSTM-type network that approximates the manifold. They validate the approach on two high-dimensional problems—the Reaction Wheel Pendulum and the parabolic Allen-Cahn equation—demonstrating real-time control performance and substantial improvements over traditional methods, with the method remaining scalable to high-dimensional systems.

Abstract

For an infinite-horizon control problem, the optimal control can be represented by the stable manifold of the characteristic Hamiltonian system of Hamilton-Jacobi-Bellman (HJB) equation in a semiglobal domain. In this paper, we first theoretically prove that if an approximation is sufficiently close to the exact stable manifold of the HJB equation in a certain sense, then the control derived from this approximation stabilizes the system and is nearly optimal. Then, based on the theoretical result, we propose a deep learning algorithm to approximate the stable manifold and compute optimal feedback control numerically. The algorithm relies on adaptive data generation through finding trajectories randomly within the stable manifold. Such kind of algorithm is grid-free basically, making it potentially applicable to a wide range of high-dimensional nonlinear systems. We demonstrate the effectiveness of our method through two examples: stabilizing the Reaction Wheel Pendulums and controlling the parabolic Allen-Cahn equation.

Paper Structure

This paper contains 16 sections, 5 theorems, 37 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The stable manifolds of the HJB equations
  3. Asymptotic analysis of approximate closed-loop system
  4. Algorithm
  5. Architecture of the NN
  6. The algorithm
  7. Key techniques of the algorithm
  8. Generating trajectories by two-point BVP and IVP
  9. Sampling along trajectories
  10. Adaptive data generation
  11. Monte Carlo test of the NN
  12. Rescaling
  13. Application to the Reaction Wheel Pendulums
  14. Application to optimal control for the parabolic Allen-Cahn equation
  15. Conclusion
  16. ...and 1 more sections

Key Result

Theorem 2.1

Assume that $f,q, R$ satisfy conditions $(C_1-C_2)$. Then the stable manifold of e:Hamiltonian-flow through $(x,p)=(0,0)$ is a smooth submanifold of dimension $n$ in $\mathbb R^{2n}$. Moreover, in a neighborhood of $(0,0)$, this submanifold is the graph $\Lambda_V$. In particular, there exist $\delt

Figures (5)

  • Figure 1: The Reaction Wheel Pendulum
  • Figure 2: Simulations with initial states at hanging position based on first round training (left subfigure) and second round training (right subfigure).
  • Figure 3: Closed-loop stabilizing trajectories of the Reaction Wheel Pendulum with some initial positions $x_0$ after the second round training.
  • Figure 4: Sample closed-loop trajectory at $x_0=(\pi, 0, 0)$ after the second round of training without rescaling at the beginning.
  • Figure 5: The dynamics of the NN controlled system at some initial states.

Theorems & Definitions (12)

  • Theorem 2.1
  • Remark 2.1
  • Proposition 2.1
  • Theorem 3.1
  • proof
  • Remark 3.1
  • Corollary 3.1
  • proof
  • Theorem 3.2
  • Remark 4.1
  • ...and 2 more