Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Optimization with Differentiable MPC: Convergence Analysis under Uncertainty

Riccardo Zuliani, Efe C. Balta, John Lygeros

Abstract

Model-based policy optimization is a well-established framework for designing reliable and high-performance controllers across a wide range of control applications. Recently, this approach has been extended to model predictive control policies, where explicit dynamical models are embedded within the control law. However, the performance of the resulting controllers, and the convergence of the associated optimization algorithms, critically depends on the accuracy of the models. In this paper, we demonstrate that combining gradient-based policy optimization with recursive system identification ensures convergence to an optimal controller design and showcase our finding in several control examples.

Policy Optimization with Differentiable MPC: Convergence Analysis under Uncertainty

Abstract

Model-based policy optimization is a well-established framework for designing reliable and high-performance controllers across a wide range of control applications. Recently, this approach has been extended to model predictive control policies, where explicit dynamical models are embedded within the control law. However, the performance of the resulting controllers, and the convergence of the associated optimization algorithms, critically depends on the accuracy of the models. In this paper, we demonstrate that combining gradient-based policy optimization with recursive system identification ensures convergence to an optimal controller design and showcase our finding in several control examples.
Paper Structure (20 sections, 15 theorems, 77 equations, 10 figures, 2 tables, 4 algorithms)

This paper contains 20 sections, 15 theorems, 77 equations, 10 figures, 2 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem formulation
  4. Proposed method
  5. Learning the true model
  6. Computing sensitivities using backpropagation
  7. The Complete Algorithm
  8. Convergence analysis
  9. Special Cases and Extensions
  10. Certainty equivalence
  11. Convergence with imperfect model learning
  12. Simulation Results
  13. Random Linear Systems
  14. Nonlinear Quadcopter
  15. Lateral Control of an Autonomous Vehicle
  16. ...and 5 more sections

Key Result

Theorem 1

Under ass:sys_idass:pe, for any confidence level $\delta\in(0,1)$, the true parameter $\theta$ belongs to the set $\Theta^k:=\{ \theta: \|\theta-\theta^k\|_{A^k} \leq c_k \}$ for all $k\in\mathbb{N}$ with probability at least $1-\delta$, where $A^k$ and $\theta^k$ are computed in eq:LS, and $c_k$ in and $\lim_{k \to \infty} \tilde{c}_k=0$.

Figures (10)

  • Figure 1: Closed-loop optimization algorithm. Observe that the nominal model $\theta^k$ in iteration $k$ need not match the prediction model $\tilde{\theta}^k$ used by the MPC.
  • Figure 2: Closed-loop optimization algorithm with certainty equivalence. The MPC prediction model $\tilde{\theta}^k$ now matches nominal model $\theta^k$.
  • Figure 3: Cost ($y$-axis) and relative suboptimality (typed number in columns) of the MPC trained with \ref{['alg:main']} (blue) against the performance of the omniscient controller (yellow) and untrained algorithm (gray), on ten randomly generated linear systems. The $+xx.xx$ number in the grey section of any column indicates that the untrained controller performed $xx.xx$ times worse than the trained controller in that experiment. The $-0.yy$ number in the yellow section indicates that the tuned controller performed $0.yy$ times worse than the omniscient controller.
  • Figure 4: Cost ($y$-axis) and relative suboptimality (typed number in columns) of the MPC trained with \ref{['alg:main_CE']} (blue) against the performance of the omniscient controller (yellow) and untrained algorithm (gray), on ten randomly generated linear systems.
  • Figure 5: Angle (top) and position (bottom) trajectories for the untrained / trained / best algorithm. In the position plot, the untrained trajectories are the dashed lines, the trained ones are solid lines, and the optimal ones are the dash-dotted lines with square markers ddded since they fall mostly below the solid lines.
  • ...and 5 more figures

Theorems & Definitions (32)

  • Theorem 1
  • proof
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Theorem 2
  • proof
  • ...and 22 more