No-Regret Model Predictive Control with Online Learning of Koopman Operators

TL;DR

The paper tackles control of nonlinear systems with unknown residual dynamics by modeling these disturbances with a Koopman operator and integrating online learning with model predictive control. It introduces a KoMpan-MPC algorithm that updates the Koopman-based disturbance model online while solving MPC, with a formal dynamic-regret guarantee showing sublinear regret and asymptotic convergence to an optimal non-causal controller. The approach is validated in physics-based cart-pole simulations under substantial model parameter uncertainty, where it outperforms nominal MPC and baselines based on Gaussian processes or random Fourier features. The work advances real-time adaptive MPC for nonlinear systems by providing a principled no-regret framework and practical online learning of disturbances via Koopman representations.

Abstract

We study a problem of simultaneous system identification and model predictive control of nonlinear systems. Particularly, we provide an algorithm for systems with unknown residual dynamics that can be expressed by Koopman operators. Such residual dynamics can model external disturbances and modeling errors, such as wind and wave disturbances to aerial and marine vehicles, or inaccurate model parameters. The algorithm has finite-time near-optimality guarantees and asymptotically converges to the optimal non-causal controller. Specifically, the algorithm enjoys sublinear \textit{dynamic regret}, defined herein as the suboptimality against an optimal clairvoyant controller that knows how the unknown dynamics will adapt to its states and actions. To this end, we assume the algorithm is given Koopman observable functions such that the unknown dynamics can be approximated by a linear dynamical system. Then, it employs model predictive control based on the current learned model of the unknown residual dynamics. This model is updated online using least squares in a self-supervised manner based on the data collected while controlling the system. We validate our algorithm in physics-based simulations of a cart-pole system aiming to maintain the pole upright despite inaccurate model parameters.

Figures (3)

• Figure 1: Overview of Pipeline for Model Predictive Control with Online Learning of Koopman Operator. The pipeline is composed of two interacting modules: (i) a model predictive control (MPC) module, and (ii) an online Koopman learning module with predefined Koopman observable functions. The MPC module uses the estimated unknown dynamics from the Koopman learning module to calculate the next control input. Given the control input and the observed new state, the online Koopman learning module then updates the estimate of the unknown dynamics.
• Figure 2: Simulation Results of the Cart-Pole Stabilization Experiment under $25\%$ Inaccurate Model Parameters. (a) Average stabilization error over $20$ runs with random initialization. (b) Sample trajectory. The results demonstrate that \ref{['alg:MPC']} (Koopman-MPC) achieves the fastest stabilization of the system among all tested algorithms. (c) Estimation error of the residual dynamics. The results demonstrate that the quick convergence of online learning of the Koopman operator with appropriately chosen observables.
• Figure 3: Sample Trajectory of the Cart-Pole Stabilization Experiment under $45\%$ Inaccurate Model Parameters.\ref{['alg:MPC']} (Koopman-MPC) successfully achieves stabilization of the system despite $45\%$ inaccuracy of the nominal model, while GP-MPC and RFF-MPC fail.

Theorems & Definitions (7)

• Definition 1: Value Function grimm2005model
• Remark 1: Approximation Error
• Definition 2: Dynamic Regret
• Remark 2: Adaptivity of $h$
• Proposition 1: Regret Bound of Online Least-Squares Estimation hazan2016introduction
• Remark 3: Combination with Offline Learned Koopman Operator
• Theorem 1: No-Regret