A Primal-dual hybrid gradient method for solving optimal control problems and the corresponding Hamilton-Jacobi PDEs

TL;DR

This work recasts optimal control problems and their Hamilton-Jacobi PDEs as saddle-point problems solved by a preconditioned PDHG algorithm with time-implicit discretization, enabling first-order accuracy and unconditional stability. It demonstrates broad applicability to Hamiltonians that are non-smooth and time-space dependent, and extends naturally to viscous HJ PDEs and stochastic control problems with diffusion. The method yields simple, parallelizable pointwise updates and reveals connections to mean-field games through time-reversed formulations. Empirical results in 1D and 2D settings validate the approach across quadratic and 1-homogeneous Hamiltonians, with both deterministic and stochastic variants, and highlight the trade-offs between solution smoothness and control structure (e.g., bang-bang controls). The framework offers a flexible, scalable alternative to traditional grid-based solvers, with potential for integration into higher-order schemes and learning-based pipelines.

Abstract

Optimal control problems are crucial in various domains, including path planning, robotics, and humanoid control, demonstrating their broad applicability. The connection between optimal control and Hamilton-Jacobi (HJ) partial differential equations (PDEs) underscores the need for solving HJ PDEs to address these control problems effectively. While numerous numerical methods exist for tackling HJ PDEs across different dimensions, this paper introduces an innovative optimization-based approach that reformulates optimal control problems and HJ PDEs into a saddle point problem using a Lagrange multiplier. Our method, based on the preconditioned primal-dual hybrid gradient (PDHG) method, offers a solution to HJ PDEs with first-order accuracy and numerical unconditional stability, enabling larger time steps and avoiding the limitations of explicit time discretization methods. Our approach has ability to handle a wide variety of Hamiltonian functions, including those that are non-smooth and dependent on time and space, through a simplified saddle point formulation that facilitates easy and parallelizable updates. Furthermore, our framework extends to viscous HJ PDEs and stochastic optimal control problems, showcasing its versatility. Through a series of numerical examples, we demonstrate the method's effectiveness in managing diverse Hamiltonians and achieving efficient parallel computation, highlighting its potential for wide-ranging applications in optimal control and beyond.

Figures (10)

• Figure 1: Visualization of the solution for the one-dimensional scenario discussed in Section \ref{['sec:eg1']}, using $n_t = 41$ and $n_x = 160$ grid points. Figures (a) and (b) showcase the level sets of the solution $\varphi$ to the HJ PDE \ref{['eqt:cont_HJ_initial']}, along with the corresponding function $\alpha$ from \ref{['eqt:feedback_initial']}, which represents the time reversal of the feedback control function. Figures (c) and (d) depict several optimal paths $s\mapsto \gamma^*(s)$ and their associated open-loop optimal controls $s\mapsto \alpha^*(s)$. These paths and control trajectories are the solutions to the optimal control problem \ref{['eqt:oc_problem']}, each beginning from a unique initial condition $x$.
• Figure 2: Depiction of the two-dimensional solution as discussed in Section \ref{['sec:eg1']}, utilizing $n_t = 41$ and $n_x = n_y = 160$ grid points. Figure (a) illustrates the level sets of the solution $\varphi(\cdot, t)$ to the HJ PDE \ref{['eqt:cont_HJ_initial']} at different times $t$. Figures (b) and (c) show the first and second components, respectively, of the associated function $\alpha(\cdot, t)$ from \ref{['eqt:feedback_initial']} at various times $t$, which depict the time reversal of the feedback control function. Figures (d) and (e) present several optimal trajectories $s\mapsto \gamma^*(s)$ along with their corresponding open-loop optimal controls $s\mapsto \alpha^*(s)$. These trajectories and control strategies solve the optimal control problem specified in \ref{['eqt:oc_problem']}, starting from distinct initial conditions $x$. Notably, both $\gamma^*$ and $\alpha^*$ take values in $\mathbb{R}^2$. For visualization, they are plotted within the spatial domain, excluding the time dimension for clarity.
• Figure 3: Visualization of the solution for the one-dimensional scenario discussed in Section \ref{['sec:eg2']}, using $n_t = 41$ and $n_x = 160$ grid points. Figures (a) and (b) showcase the level sets of the solution $\varphi$ to the HJ PDE \ref{['eqt:cont_HJ_initial']}, along with the corresponding function $\alpha$ from \ref{['eqt:feedback_initial']}, which represents the time reversal of the feedback control function. Figures (c) and (d) depict several optimal paths $s\mapsto \gamma^*(s)$ and their associated open-loop optimal controls $s\mapsto \alpha^*(s)$. These paths and control trajectories are the solutions to the optimal control problem \ref{['eqt:oc_problem']}, each beginning from a unique initial condition $x$.
• Figure 4: Depiction of the two-dimensional solution as discussed in Section \ref{['sec:eg2']}, utilizing $n_t = 41$ and $n_x = n_y = 160$ grid points. Figure (a) illustrates the level sets of the solution $\varphi(\cdot, t)$ to the HJ PDE \ref{['eqt:cont_HJ_initial']} at different times $t$. Figures (b) and (c) show the first and second components, respectively, of the associated function $\alpha(\cdot, t)$ from \ref{['eqt:feedback_initial']} at various times $t$, which depict the time reversal of the feedback control function. Figures (d) and (e) present several optimal trajectories $s\mapsto \gamma^*(s)$ along with their corresponding open-loop optimal controls $s\mapsto \alpha^*(s)$. These trajectories and control strategies solve the optimal control problem specified in \ref{['eqt:oc_problem']}, starting from distinct initial conditions $x$. Notably, both $\gamma^*$ and $\alpha^*$ take values in $\mathbb{R}^2$. For visualization, they are plotted within the spatial domain, excluding the time dimension for clarity.
• Figure 5: Visualization of the solution for example discussed in Section \ref{['sec:numerical_newton_det']}, using $n_t = 41$, $n_x = 160$, and $n_y=80$ grid points. Figures (a) and (b) showcase the level sets of the solution $\varphi$ to the HJ PDE \ref{['eqt:cont_HJ_initial']}, along with the corresponding function $\alpha$ from \ref{['eqt:feedback_initial']}, which represents the time reversal of the feedback control function. Figures (c), (d), (e) depict the first and second components of several optimal paths $s\mapsto \gamma^*(s)$ and their associated open-loop optimal controls $s\mapsto \alpha^*(s)$. These paths and control trajectories are the solutions to the optimal control problem \ref{['eqt:oc_problem']}, each beginning from a unique initial condition $x$.

Theorems & Definitions (10)

• remark thmcounterremark: Convergence of the Proposed Algorithm
• remark thmcounterremark: Details on Implementation
• remark thmcounterremark: Relation to MFGs
• remark thmcounterremark: Comparison with the Approach in meng2023primal
• Remark A.1: Connection of the PDHG updates above with our proposed updates
• Remark A.2: PDHG convergence for discretized problems
• Remark B.1
• Remark B.2
• Remark B.3
• Remark B.4