Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pontryagin Optimal Control via Neural Networks

Chengyang Gu, Hui Xiong, Yize Chen

TL;DR

The paper tackles optimal control for systems with unknown, nonlinear dynamics by fusing neural network surrogates with Pontryagin's Maximum Principle. It introduces the NN-PMP-Gradient framework, which learns a neural model $\mathbf{x}_{t+1}=f_{NN}(\mathbf{x}_t,\mathbf{u}_t;\theta)$ and solves the PMP conditions via a differentiable, gradient-based procedure. The authors prove convergence results under convexity and Lipschitz conditions and demonstrate superior sample efficiency and performance compared to model-free PPO and model-based RS-MPC across five tasks, including LQR, Battery, Pendulum, Swimmer, and HalfCheetah. This approach offers a practical, interpretable, and safer model-based control strategy for real-world systems with unknown dynamics, with open-source code enabling replication and extension.

Abstract

Solving real-world optimal control problems are challenging tasks, as the complex, high-dimensional system dynamics are usually unrevealed to the decision maker. It is thus hard to find the optimal control actions numerically. To deal with such modeling and computation challenges, in this paper, we integrate Neural Networks with the Pontryagin's Maximum Principle (PMP), and propose a sample efficient framework NN-PMP-Gradient. The resulting controller can be implemented for systems with unknown and complex dynamics. By taking an iterative approach, the proposed framework not only utilizes the accurate surrogate models parameterized by neural networks, it also efficiently recovers the optimality conditions along with the optimal action sequences via PMP conditions. Numerical simulations on Linear Quadratic Regulator, energy arbitrage of grid-connected lossy battery, control of single pendulum, and two MuJoCo locomotion tasks demonstrate our proposed NN-PMP-Gradient is a general and versatile computation tool for finding optimal solutions. And compared with the widely applied model-free and model-based reinforcement learning (RL) algorithms, our NN-PMP-Gradient achieves higher sample-efficiency and performance in terms of control objectives.

Pontryagin Optimal Control via Neural Networks

TL;DR

The paper tackles optimal control for systems with unknown, nonlinear dynamics by fusing neural network surrogates with Pontryagin's Maximum Principle. It introduces the NN-PMP-Gradient framework, which learns a neural model and solves the PMP conditions via a differentiable, gradient-based procedure. The authors prove convergence results under convexity and Lipschitz conditions and demonstrate superior sample efficiency and performance compared to model-free PPO and model-based RS-MPC across five tasks, including LQR, Battery, Pendulum, Swimmer, and HalfCheetah. This approach offers a practical, interpretable, and safer model-based control strategy for real-world systems with unknown dynamics, with open-source code enabling replication and extension.

Abstract

Solving real-world optimal control problems are challenging tasks, as the complex, high-dimensional system dynamics are usually unrevealed to the decision maker. It is thus hard to find the optimal control actions numerically. To deal with such modeling and computation challenges, in this paper, we integrate Neural Networks with the Pontryagin's Maximum Principle (PMP), and propose a sample efficient framework NN-PMP-Gradient. The resulting controller can be implemented for systems with unknown and complex dynamics. By taking an iterative approach, the proposed framework not only utilizes the accurate surrogate models parameterized by neural networks, it also efficiently recovers the optimality conditions along with the optimal action sequences via PMP conditions. Numerical simulations on Linear Quadratic Regulator, energy arbitrage of grid-connected lossy battery, control of single pendulum, and two MuJoCo locomotion tasks demonstrate our proposed NN-PMP-Gradient is a general and versatile computation tool for finding optimal solutions. And compared with the widely applied model-free and model-based reinforcement learning (RL) algorithms, our NN-PMP-Gradient achieves higher sample-efficiency and performance in terms of control objectives.
Paper Structure (14 sections, 2 theorems, 42 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 14 sections, 2 theorems, 42 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Problem Statement
  4. Pontyragin Maximum Principle
  5. Model-based Optimal Controller Design
  6. System Identification with Neural Networks
  7. PMP-Gradient Controller Design
  8. Numerical Examples
  9. Simulated Control Tasks
  10. Numerical Results
  11. Conclusion and Discussion
  12. Convergence Proof of PMP-Gradient Controller
  13. Settings of Neural-PMP for Problems with State and Action Bounds
  14. Detailed Settings of Experiment Environments

Key Result

Lemma 1

Implementing Eq. equ:u_iter is equivalent to taking gradient descent step $\mathbf{u}_{t}^{(k+1)} =\mathbf{u}_{t}^{(k)}- \eta \frac{\partial J}{ \partial \mathbf{u}_{t}^{(k)}}$.

Figures (3)

  • Figure 1: Schematic of our proposed model learning and control framework based on PMP conditions.
  • Figure 2: Sample-Performance (Cost/Reward) curves on (a) LQR (C); (b) Battery (C); (c) Pendulum (C); (d) Swimmer (R); (e) HalfCheetah (R).
  • Figure 3: Battery state trajectory obtained by Neural-PMP is the only one not exceeding state bounds.

Theorems & Definitions (4)

  • Lemma 1
  • Theorem 2
  • proof
  • proof