Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence Analysis of the Upwind Difference Methods for Hamilton-Jacobi-Bellman Equations

Daisuke Inoue, Yuji Ito, Takahito Kashiwabara, Norikazu Saito, Hiroaki Yoshida

TL;DR

This work analyzes the upwind difference scheme for Hamilton-Jacobi-Bellman equations arising in optimal control. It shows that, under a classical (smooth) solution, the value function converges with a first-order rate in both time and space, i.e., $O(\Delta t^{\eta}+\Delta x^{\eta})$ for some $\eta\in(0,1]$, while the standard viscosity-solution theory gives $O(\sqrt{\Delta t})$ in general. The authors further establish convergence of the numerical control input in the one-dimensional setting by exploiting the link between HJB equations and scalar conservation laws, using $L^1$ convergence of spatial derivatives and epi-convergence to guarantee that the discrete minimizers converge to the true optimal input $a^*(x,t)$. Numerical experiments in 1D and 2D corroborate the theoretical rates and demonstrate the scheme’s practical reliability for control synthesis. The results provide rigorous guarantees for using upwind schemes in discrete HJB-based control and highlight an open problem of extending control-input convergence to higher dimensions.

Abstract

This paper investigates the convergence properties of the upwind difference scheme for the Hamilton--Jacobi--Bellman (HJB) equation, a central partial differential equation in optimal control theory. First, assuming the existence of a classical solution, we show that the numerical solution converges to the true solution with a first-order rate with respect to the time step. This result complements the square-root rate established in previous studies for viscosity solutions. Second, by exploiting the correspondence between HJB equations and conservation laws, we prove the convergence of the optimal control input. This analysis is crucial for practical applications where the control input is the primary quantity of interest, yet it has rarely been addressed in previous studies. Finally, we confirm the validity of our theoretical results through numerical experiments on typical control problems.

Convergence Analysis of the Upwind Difference Methods for Hamilton-Jacobi-Bellman Equations

TL;DR

This work analyzes the upwind difference scheme for Hamilton-Jacobi-Bellman equations arising in optimal control. It shows that, under a classical (smooth) solution, the value function converges with a first-order rate in both time and space, i.e., for some , while the standard viscosity-solution theory gives in general. The authors further establish convergence of the numerical control input in the one-dimensional setting by exploiting the link between HJB equations and scalar conservation laws, using convergence of spatial derivatives and epi-convergence to guarantee that the discrete minimizers converge to the true optimal input . Numerical experiments in 1D and 2D corroborate the theoretical rates and demonstrate the scheme’s practical reliability for control synthesis. The results provide rigorous guarantees for using upwind schemes in discrete HJB-based control and highlight an open problem of extending control-input convergence to higher dimensions.

Abstract

This paper investigates the convergence properties of the upwind difference scheme for the Hamilton--Jacobi--Bellman (HJB) equation, a central partial differential equation in optimal control theory. First, assuming the existence of a classical solution, we show that the numerical solution converges to the true solution with a first-order rate with respect to the time step. This result complements the square-root rate established in previous studies for viscosity solutions. Second, by exploiting the correspondence between HJB equations and conservation laws, we prove the convergence of the optimal control input. This analysis is crucial for practical applications where the control input is the primary quantity of interest, yet it has rarely been addressed in previous studies. Finally, we confirm the validity of our theoretical results through numerical experiments on typical control problems.
Paper Structure (13 sections, 7 theorems, 74 equations, 12 figures)

This paper contains 13 sections, 7 theorems, 74 equations, 12 figures.

Table of Contents

  1. Introduction
  2. Review on Hamilton--Jacobi--Bellman Equation and Upwind Difference Scheme
  3. HJB Equation
  4. Upwind Difference Scheme
  5. Convergence Results
  6. Proof of Results
  7. Convergence of the Variable $v$
  8. Convergence of the Control Input Function $a^*$
  9. Numerical Experiments
  10. One-dimensional LQR problem
  11. One-dimensional sparse control problem
  12. Two-dimensional problem
  13. Conclusion

Key Result

Proposition 1

Under Assumption assmp:HJB, the HJB equation eq:HJB has a unique viscosity solution. In addition, this solution coincides with the value function defined in eq:value_function.

Figures (12)

  • Figure 1: Time response for the variable $v$.
  • Figure 2: Time response for the variable $a$.
  • Figure 3: Error of variable $v$ for various $\Delta x$.
  • Figure 4: Error of variable $a$ for various $\Delta x$.
  • Figure 5: Time response for the variable $v$.
  • ...and 7 more figures

Theorems & Definitions (19)

  • Proposition 1: Theorem 7.1 and Theorem 9.1 of chapter 2 in Fleming2006Controlled
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Theorem 1
  • Example 1
  • Theorem 2
  • proof : Proof of Theorem \ref{['thm:convergence']}
  • Remark 5
  • ...and 9 more