Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Learning-Based Predictive Control for Nonlinear System

Yuanqing Zhang, Huanshui Zhang

TL;DR

The paper addresses predictive control for nonlinear systems without explicit dynamics by fusing reinforcement learning with model-free online optimization. An RL-based LPC solver is embedded within an OCP framework to deliver real-time, Hessian-free updates and guarantee super-linear convergence, even when Hessians are singular. The approach is validated across linear, nonlinear, and trajectory-tracking tasks, demonstrating stability, faster convergence, and practical online feasibility with runtimes around a few hundredths of a second per horizon. This data-driven, Hessian-robust LPC offers a scalable alternative to traditional MPC when models are unavailable or costly to identify, with clear advantages in computation efficiency and robustness.

Abstract

In this paper, we propose an online learning-based predictive control (LPC) approach designed for nonlinear systems that lack explicit system dynamics. Unlike traditional model predictive control (MPC) algorithms that rely on known system models to optimize controller outputs, our proposed algorithm integrates a reinforcement learning component to learn optimal policies in real time from the offline dataset and real-time data. Additionally, an optimal control problem (OCP)-based optimization framework is incorporated to enhance real-time computational efficiency while ensuring stability during online operation. Moreover, we rigorously establish the super-linear convergence properties of the algorithm. Finally, extensive simulations are performed to evaluate the feasibility and effectiveness of the proposed approach.

Online Learning-Based Predictive Control for Nonlinear System

TL;DR

The paper addresses predictive control for nonlinear systems without explicit dynamics by fusing reinforcement learning with model-free online optimization. An RL-based LPC solver is embedded within an OCP framework to deliver real-time, Hessian-free updates and guarantee super-linear convergence, even when Hessians are singular. The approach is validated across linear, nonlinear, and trajectory-tracking tasks, demonstrating stability, faster convergence, and practical online feasibility with runtimes around a few hundredths of a second per horizon. This data-driven, Hessian-robust LPC offers a scalable alternative to traditional MPC when models are unavailable or costly to identify, with clear advantages in computation efficiency and robustness.

Abstract

In this paper, we propose an online learning-based predictive control (LPC) approach designed for nonlinear systems that lack explicit system dynamics. Unlike traditional model predictive control (MPC) algorithms that rely on known system models to optimize controller outputs, our proposed algorithm integrates a reinforcement learning component to learn optimal policies in real time from the offline dataset and real-time data. Additionally, an optimal control problem (OCP)-based optimization framework is incorporated to enhance real-time computational efficiency while ensuring stability during online operation. Moreover, we rigorously establish the super-linear convergence properties of the algorithm. Finally, extensive simulations are performed to evaluate the feasibility and effectiveness of the proposed approach.

Paper Structure

This paper contains 17 sections, 3 theorems, 45 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Formulation of Predictive Control Problem
  4. Reinforcement Learning
  5. OCP Method
  6. Online LPC Controller Design
  7. The Formation of LPC
  8. RL-Based LPC Solver
  9. OCP Method Based Online RL Process
  10. Design of the Critic Networks
  11. Design of the Actor Networks
  12. Algorithmic Process of Online LPC
  13. Convergence Analysis
  14. Experiments
  15. Linear System Control
  16. ...and 2 more sections

Key Result

Lemma 1

Let $\textbf{z}_k$ be a sequence generated by an iterative method converging to a solution $z_*$. We say that the sequence $\textbf{z}_k$ converges super-linearly to $z_*$ if there exists a positive integer $p\ge1$ such that for sufficiently large $k$,

Figures (8)

  • Figure 1: Online LPC Structure
  • Figure 2: State trajectory for linear system
  • Figure 3: Control input trajectory for linear system
  • Figure 4: Convergence analysis of critic and actors’ weights. (a) Trajectory of critic’s weights with proposed algorithm. (b) Trajectory of critic’s weights with traditional algorithm. (c) Trajectory of actor’s weights with proposed algorithm. (d) Trajectory of actor’s weights with traditional algorithm.
  • Figure 5: State trajectory for non-linear system
  • ...and 3 more figures

Theorems & Definitions (11)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • Remark 6
  • Lemma 1
  • Lemma 2
  • Theorem 1
  • proof
  • ...and 1 more