Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learning

Jae Yong Lee, Yeoneung Kim

TL;DR

This paper incorporates DeepONet with a recently developed policy iteration scheme to numerically solve optimal control problems and the corresponding Hamilton--Jacobi--Bellman (HJB) equations.

Abstract

The framework of deep operator network (DeepONet) has been widely exploited thanks to its capability of solving high dimensional partial differential equations. In this paper, we incorporate DeepONet with a recently developed policy iteration scheme to numerically solve optimal control problems and the corresponding Hamilton--Jacobi--Bellman (HJB) equations. A notable feature of our approach is that once the neural network is trained, the solution to the optimal control problem and HJB equations with different terminal functions can be inferred quickly thanks to the unique feature of operator learning. Furthermore, a quantitative analysis of the accuracy of the algorithm is carried out via comparison principles of viscosity solutions. The effectiveness of the method is verified with various examples, including 10-dimensional linear quadratic regulator problems (LQRs).

Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learning

TL;DR

This paper incorporates DeepONet with a recently developed policy iteration scheme to numerically solve optimal control problems and the corresponding Hamilton--Jacobi--Bellman (HJB) equations.

Abstract

The framework of deep operator network (DeepONet) has been widely exploited thanks to its capability of solving high dimensional partial differential equations. In this paper, we incorporate DeepONet with a recently developed policy iteration scheme to numerically solve optimal control problems and the corresponding Hamilton--Jacobi--Bellman (HJB) equations. A notable feature of our approach is that once the neural network is trained, the solution to the optimal control problem and HJB equations with different terminal functions can be inferred quickly thanks to the unique feature of operator learning. Furthermore, a quantitative analysis of the accuracy of the algorithm is carried out via comparison principles of viscosity solutions. The effectiveness of the method is verified with various examples, including 10-dimensional linear quadratic regulator problems (LQRs).
Paper Structure (16 sections, 4 theorems, 55 equations, 5 figures, 1 algorithm)

This paper contains 16 sections, 4 theorems, 55 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Organization of paper
  3. Notations
  4. Preliminary
  5. Optimal control and Hamilton--Jacobi--Bellman equation
  6. Semi-discrete iterative scheme
  7. Physics-Informed DeepONet (PI-DeepONet)
  8. Main Results and Methodology
  9. Solving optimal control by using PI-DeepONet iteratively
  10. Error analysis
  11. Experimental result
  12. Synthetic case 2D
  13. LQR with a compact control set
  14. $(d,m)=(5,3)$ with $A \in \mathbb{R}^{5\times 5}$ and $B \in \mathbb{R}^{5\times 3}$.
  15. $(d,m)=(10,5)$ with $A \in \mathbb{R}^{10\times 5}$ and $B \in \mathbb{R}^{5\times 5}$.
  16. ...and 1 more sections

Key Result

Theorem 1

Let $N \geq \max\{1,\|f\|_\infty/2\}$ and $h \in (0,1)$ be given and assume that (A1) and (A2) are enforced. For $V_n^h$ solving eq:semidiscret, we have that where $V^h$ is a viscosity solution to Here, $H(t, x, p):=\inf_{{\boldsymbol{u}}\in U} \left\{ p\cdot f(t,x, {\boldsymbol{u}})+L(t, x, {\boldsymbol{u}})\right\}$. Furthermore, for the unique viscosity solution $V$ of eq:hj, we deduce that

Figures (5)

  • Figure 1: PI-DeepONet framework.
  • Figure 2: Our framework: Hamilton-Jacobi based policy-iteration via PI-DeepONet
  • Figure 3: Plots of $V_M^h(t,x)$ with $t \in [0,1]$ fixing $x[2]=-0.5$ (left). Optimal controls corresponding to different initial points of $x$ with $x[1] \in [-1.5,1.5]$ with $x[2]=-0.5$ (middle). Optimal state trajectories associated with the controls computed (right).
  • Figure 4: Viscoisity solutions, state, and control trajectories for LQR problem with $(d,m)=(5,3)$. Value functions $V_M^h$ with different terminal functions $g(x)$ for the LQR problem (top). Optimal trajectories (middle) and controls (bottom) associated with different initial values of $x_0= x^{\text{new}}$.
  • Figure 5: Viscoisity solutions, state, and control trajectories for LQR problem with $(d,m)=(5,3)$. Value functions $V_M^h$ with different terminal functions $g(x)$ for the LQR problem (top). Optimal trajectories (middle) and controls (bottom) associated with different initial values of $x_0= x^{\text{new}}$.

Theorems & Definitions (8)

  • Theorem 1
  • Proposition 1: Uniform boundedness
  • proof
  • Definition 1
  • Theorem 2: Stability of $v_n^h$
  • proof
  • Corollary 1
  • proof