Ensemble based Closed-Loop Optimal Control using Physics-Informed Neural Networks

TL;DR

This work develops an ensemble Physics-Informed Neural Network (PINN) framework to solve infinite-horizon, time-invariant optimal control problems by learning the optimal cost-to-go $\mathcal{J}$ and its costate $\lambda=\nabla_x \mathcal{J}$ from the Hamilton-Jacobi-Bellman equation. It introduces three ensemble-control strategies—individual, mean, and outlier-exclusion mean—showing that an ensemble with robust outlier handling can stabilize a two-state nonlinear affine system under noise and varied initial conditions without stabilizer terms. The methodology combines warm-start data with a dedicated HJB-PINN loss that enforces boundary conditions and PDE residuals, yielding accurate adjoints $\lambda$ and control signals $u^*$. The results demonstrate that the ensemble approach can reproduce analytical solutions and maintain stable closed-loop behavior, offering a scalable path toward more complex nonlinear control problems. The work contributes a practical, data-efficient framework for closed-loop control under uncertainty with potential applications in robotics and engineered systems where long-horizon optimality is essential.

Abstract

The objective of designing a control system is to steer a dynamical system with a control signal, guiding it to exhibit the desired behavior. The Hamilton-Jacobi-Bellman (HJB) partial differential equation offers a framework for optimal control system design. However, numerical solutions to this equation are computationally intensive, and analytical solutions are frequently unavailable. Knowledge-guided machine learning methodologies, such as physics-informed neural networks (PINNs), offer new alternative approaches that can alleviate the difficulties of solving the HJB equation numerically. This work presents a multistage ensemble framework to learn the optimal cost-to-go, and subsequently the corresponding optimal control signal, through the HJB equation. Prior PINN-based approaches rely on a stabilizing the HJB enforcement during training. Our framework does not use stabilizer terms and offers a means of controlling the nonlinear system, via either a singular learned control signal or an ensemble control signal policy. Success is demonstrated in closed-loop control, using both ensemble- and singular-control, of a steady-state time-invariant two-state continuous nonlinear system with an infinite time horizon, accounting of noisy, perturbed system states and varying initial conditions.

Figures (7)

• Figure 1: Validation of system dynamics for initial conditions $x_1$, = $x_2$ = 10.0. Panel (a) shows the system dynamics using the outlier exclusion ensemble control signal policy. Panel (b) shows the system dynamics using the outlier inclusion ensemble control signal policy. Panel (c) shows the system dynamics using individual control signal policies, as well as the system dynamics under zero/no control signal policy.
• Figure 2: Mean Squared Error (MSE) plots for the derived Lambda ($\hat{\lambda}^e$) and the calculated optimal control signal ($\hat{u}^e$). Panels (a) and (b) show the MSE between the analytical Lambda ($\lambda^*$) and the differentiated Lambda ($\hat{\lambda}^e$) from the learned Cost-to-Go ($\widehat{\mathcal{J}}^e$) with respect to the system state ($x$), for a given model among the ensemble. Panel (a) shows the MSE($\lambda_1^*, \hat{\lambda}_1^e$). Panel (b) shows the MSE($\lambda_2^*, \hat{\lambda}_2^e$). Panel (c) shows the MSE between the analytical control signal ($u^*$) and the differentiated optimal control signal ($\hat{u}^e$), for a given model in the ensemble. $\hat{u}^e$ is calculated via the differentiated Lambda ($\hat{\lambda}^e$) from the learned Cost-to-Go ($\widehat{\mathcal{J}}^e$) with respect to the system state ($x$).
• Figure 3: Three-dimensional reconstruction and comparison between the reference analytical solution and the learned network solution for both the adjoint ($\lambda_1$ and $\lambda_2$) and control ($u$).Panel (a) shows the analytical solution $\lambda_1^*$, whereas panel (b) shows the learned solution of $\hat{\lambda}_1^e$. Panel (c) shows the analytical solution of $\lambda_2^*$, whereas panel (d) shows the learned solution of $\hat{\lambda}_2^e$. Panel (e) shows the analytical solution of $u^*$, whereas panel (f) shows the learned solution of $\hat{u}^e$.
• Figure 4: Three-dimensional reconstruction of the Hamiltonian from the network's derived adjoint ($\hat{\lambda}^e$) and calculated optimal control ($\hat{u}^e$) . The reference solution of the Hamiltonian is $H^*(x, u, \lambda) = 0$.
• Figure :

Theorems & Definitions (4)

• Remark 1: Key distinctions between Cost-to-Go ($\mathcal{J}$) and Value ($V$) functions
• Definition 1: Bellman’s Principle of Optimality (PO)
• Remark 2: Dependent variables of the Cost-to-Go $\mathcal{J}$ before and after optimization
• Remark 3: Steady-state HJB