Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AC4MPC: Actor-Critic Reinforcement Learning for Nonlinear Model Predictive Control

Rudolf Reiter, Andrea Ghezzi, Katrin Baumgärtner, Jasper Hoffmann, Robert D. McAllister, Moritz Diehl

TL;DR

The paper addresses the challenge of achieving high-performance control for nonlinear systems by combining model predictive control (MPC) with actor-critic reinforcement learning (RL).AC4MPC uses a trained RL critic to approximate the terminal value and an actor rollout to initialize MPC, and augments RTI with parallel solvers to enable real-time operation.The authors provide a theoretical cost-decrease guarantee that holds without requiring globally optimal MPC solutions and demonstrate its effectiveness on a snow-hill toy example and a time-optimal autonomous driving overtaking scenario.The work offers a practical framework for leveraging RL in MPC with real-time capabilities and suggests guidelines for horizon and rollout lengths and networking choices.

Abstract

\Ac{MPC} and \ac{RL} are two powerful control strategies with, arguably, complementary advantages. In this work, we show how actor-critic \ac{RL} techniques can be leveraged to improve the performance of \ac{MPC}. The \ac{RL} critic is used as an approximation of the optimal value function, and an actor roll-out provides an initial guess for primal variables of the \ac{MPC}. A parallel control architecture is proposed where each \ac{MPC} instance is solved twice for different initial guesses. Besides the actor roll-out initialization, a shifted initialization from the previous solution is used. Thereafter, the actor and the critic are again used to approximately evaluate the infinite horizon cost of these trajectories. The control actions from the lowest-cost trajectory are applied to the system at each time step. We establish that the proposed algorithm is guaranteed to outperform the original \ac{RL} policy plus an error term that depends on the accuracy of the critic and decays with the horizon length of the \ac{MPC} formulation. Moreover, we do not require globally optimal solutions for these guarantees to hold. The approach is demonstrated on an illustrative toy example and an \ac{AD} overtaking scenario.

AC4MPC: Actor-Critic Reinforcement Learning for Nonlinear Model Predictive Control

TL;DR

The paper addresses the challenge of achieving high-performance control for nonlinear systems by combining model predictive control (MPC) with actor-critic reinforcement learning (RL).AC4MPC uses a trained RL critic to approximate the terminal value and an actor rollout to initialize MPC, and augments RTI with parallel solvers to enable real-time operation.The authors provide a theoretical cost-decrease guarantee that holds without requiring globally optimal MPC solutions and demonstrate its effectiveness on a snow-hill toy example and a time-optimal autonomous driving overtaking scenario.The work offers a practical framework for leveraging RL in MPC with real-time capabilities and suggests guidelines for horizon and rollout lengths and networking choices.

Abstract

\Ac{MPC} and \ac{RL} are two powerful control strategies with, arguably, complementary advantages. In this work, we show how actor-critic \ac{RL} techniques can be leveraged to improve the performance of \ac{MPC}. The \ac{RL} critic is used as an approximation of the optimal value function, and an actor roll-out provides an initial guess for primal variables of the \ac{MPC}. A parallel control architecture is proposed where each \ac{MPC} instance is solved twice for different initial guesses. Besides the actor roll-out initialization, a shifted initialization from the previous solution is used. Thereafter, the actor and the critic are again used to approximately evaluate the infinite horizon cost of these trajectories. The control actions from the lowest-cost trajectory are applied to the system at each time step. We establish that the proposed algorithm is guaranteed to outperform the original \ac{RL} policy plus an error term that depends on the accuracy of the critic and decays with the horizon length of the \ac{MPC} formulation. Moreover, we do not require globally optimal solutions for these guarantees to hold. The approach is demonstrated on an illustrative toy example and an \ac{AD} overtaking scenario.
Paper Structure (21 sections, 4 theorems, 44 equations, 7 figures, 1 table, 3 algorithms)

This paper contains 21 sections, 4 theorems, 44 equations, 7 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contribution
  4. Outline
  5. Preliminaries
  6. Reinforcement Learning
  7. Nonlinear Model Predictive Control
  8. Actor and Critic Models for Nonlinear Model Predictive Control
  9. Basic AC4MPC Algorithm Description
  10. Cost Reduction and Performance
  11. Multiple Shooting and Real-Time Iterations for AC4MPC
  12. Parallelization
  13. Evaluation
  14. Experiments
  15. Using Neural Networks within MPC
  16. ...and 6 more sections

Key Result

Lemma 1

If Assumption as:critic holds, then in which $s^+=F(s,\kappa_N(s,\tilde{\mathbf{u}}))$ and $\tilde{\mathbf{u}}^+=\zeta(s,K_N(s,\tilde{\mathbf{u}});\hat{\pi}(\cdot))$ for all $s\in\mathbb{R}^{n_s}$ and $\tilde{\mathbf{u}}\in\mathbb{U}^{N}$.

Figures (7)

  • Figure 1: Algorithm sketch of AC4MPC-RTI. In each iteration, the actor policy is rolled out to obtain a control and state trajectory (red). After each P iterations, the parallel MPC is initialized with the policy roll-out (yellow) and, otherwise, by the shifted previous MPC solution. The active MPC is initialized with the lowest-cost trajectory, which could either be the shifted solution of its last iteration, the parallel MPC trajectory, or the policy roll-out. The cost is provided by the proposed evaluation algorithm ac4eval (green).
  • Figure 2: Force acting on the 1D vehicle due to a snowy slope and the maximum input acceleration force in the snow hill environment.
  • Figure 3: Comparison of closed-loop trajectories $S^j$ and relevant value functions for different control algorithms applied to the snow hill environment. Closed-loop trajectories are evaluated for four different starting states $\hat{s}^j$, simulating the system for 20 seconds, following the related policy. The goal state $\tilde{s}=[0,0]^\top$ can only be reached by certain algorithm variants. The first plots shows the ground truth value function $J^*$ and trajectories obtained by MPC with a sufficiently long horizon to reach the goal state and constrained to the same. The next plot shows trajectories obtained by dynamic programming (DP), which are only close to optimal due to the discretization error. The difference $\Delta J^\mathrm{dp}=J^*-J^\mathrm{dp}$ of the DP value function $J^\mathrm{dp}$ to the optimal counterpart is shown. The next two plots show simulated trajectories using the actor, as well as the critic value function difference $\Delta J^\mathrm{sac}$ to the optimal one of SAC RL after $5\cdot 10^4$ and $10^5$ iterations, respectively. The lower left plot shows the nominal MPC evaluation by initializing the trajectories at the current state and solving until full convergence. The value function $J^\mathrm{mpc}$, corresponds to the open-loop values computed by the NMPC. Next, the A4MPC, which uses the actor obtained by SAC to initialize each MPC closed-loop iteration but no terminal value function, C4MPC which uses the initial state as initial guess the critic of the SAC as terminal value function, and AC4MPC which uses both, the actor and the critic of the SAC are shown. The value functions plotted for A4MPC, C4MPC, and AC4MPC correspond to the SAC critic value $J^\mathrm{sac50}$, which is used directly in C4MPC and AC4MPC as the terminal value function, and the related policy roll-out is used in A4MPC and AC4MPC-RTI.
  • Figure 4: Comparison of average suboptimality $\rho=(J^{\{\cdot\}}-J^*)/J^*$ evaluated for closed-loop accumulated costs, corresponding to the closed-loop value functions, for different control algorithms on the snow hill environment. The proposed AC4MPC algorithm outperforms all other approaches, including the DP that is slightly sub-optimal due to the discretization error of the state and control space. In this example, using only the critic (C4MPC) or the actor (A4MPC) leads to high costs, only slightly improving the nominal MPC. Using AC4MPC has a high computational demand due to solving the optimization problem towards convergence in each iteration. Therefore, AC4MPC-RTI significantly reduces the online computation time yet slightly increases the closed-loop cost. The cost of AC4MPC-RTI is considerably lower than the RL SAC cost. The zoomed range in the upper plot is highlighted in green.
  • Figure 5: Phase plot of the AC4MPC-RTI closed-loop trajectory in the snow hill environment, starting from the state $s_0=[\shortminus5,\shortminus1]^\top$ and ending in the goal state $\tilde{s}=[0,0]^\top$. At each $P=5$ iterations, the parallel MPC is re-initialized by using the actor policy. In each iteration, the actor is also rolled out by simulation. Using a cost evaluation parameter of $\alpha=1$, the control corresponding to the lowest-cost trajectory is applied to the system. The upper plot shows whether an NMPC control was applied in the current time step (blue) or the proposed RL action (red). Additionally, green triangles indicate if, in the particular time step, the source of the output changed from either of the NMPC variants or the policy roll-out. The lower plot shows the parallel roll-outs of potentially both inactive NMPCs (orange) and the RL roll-out (red).
  • ...and 2 more figures

Theorems & Definitions (8)

  • Lemma 1: Cost decrease
  • proof
  • Theorem 2: Performance
  • proof
  • Corollary 3
  • proof
  • Corollary 4: Long-term performance
  • proof