Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning-based Control of Nonlinear Systems using Carleman Approximation: Structured and Unstructured Designs

Jishnudeep Kar, He Bai, Aranya Chakrabortty

TL;DR

This work introduces reinforcement learning for unknown nonlinear, input-affine systems by embedding the dynamics into an infinite-dimensional Carleman space, yielding a bilinear lifted model suitable for policy-iteration RL. It develops both on-policy and off-policy learning algorithms, derives a Lyapunov-based stability framework, and analyzes the impact of finite-N truncation with explicit truncation-error bounds. To address practical constraints, the authors extend the framework to structured and sparse controllers using Riccati-like equations in Carleman space and ADMM-based sparsity promotion, respectively, while preserving closed-loop stability. Numerical experiments on a second-order oscillator and a four-boat tug network illustrate near-optimal performance, faster learning than NN-based approaches, and clear trade-offs between structure, sparsity, and control performance. The proposed framework offers a scalable, data-driven, stability-guaranteed path to nonlinear RL control with tunable complexity via truncation order.

Abstract

We develop data-driven reinforcement learning (RL) control designs for input-affine nonlinear systems. We use Carleman linearization to express the state-space representation of the nonlinear dynamical model in the Carleman space, and develop a real-time algorithm that can learn nonlinear state-feedback controllers using state and input measurements in the infinite-dimensional Carleman space. Thereafter, we study the practicality of having a finite-order truncation of the control signal, followed by its closed-loop stability analysis. Finally, we develop two additional designs that can learn structured as well as sparse representations of the RL-based nonlinear controller, and provide theoretical conditions for ensuring their closed-loop stability. We present numerical examples to show how our proposed method generates closed-loop responses that are close to the optimal performance of the nonlinear plant. We also compare our designs to other data-driven nonlinear RL control methods such as those based on neural networks, and illustrate their relative advantages and drawbacks.

Reinforcement Learning-based Control of Nonlinear Systems using Carleman Approximation: Structured and Unstructured Designs

TL;DR

This work introduces reinforcement learning for unknown nonlinear, input-affine systems by embedding the dynamics into an infinite-dimensional Carleman space, yielding a bilinear lifted model suitable for policy-iteration RL. It develops both on-policy and off-policy learning algorithms, derives a Lyapunov-based stability framework, and analyzes the impact of finite-N truncation with explicit truncation-error bounds. To address practical constraints, the authors extend the framework to structured and sparse controllers using Riccati-like equations in Carleman space and ADMM-based sparsity promotion, respectively, while preserving closed-loop stability. Numerical experiments on a second-order oscillator and a four-boat tug network illustrate near-optimal performance, faster learning than NN-based approaches, and clear trade-offs between structure, sparsity, and control performance. The proposed framework offers a scalable, data-driven, stability-guaranteed path to nonlinear RL control with tunable complexity via truncation order.

Abstract

We develop data-driven reinforcement learning (RL) control designs for input-affine nonlinear systems. We use Carleman linearization to express the state-space representation of the nonlinear dynamical model in the Carleman space, and develop a real-time algorithm that can learn nonlinear state-feedback controllers using state and input measurements in the infinite-dimensional Carleman space. Thereafter, we study the practicality of having a finite-order truncation of the control signal, followed by its closed-loop stability analysis. Finally, we develop two additional designs that can learn structured as well as sparse representations of the RL-based nonlinear controller, and provide theoretical conditions for ensuring their closed-loop stability. We present numerical examples to show how our proposed method generates closed-loop responses that are close to the optimal performance of the nonlinear plant. We also compare our designs to other data-driven nonlinear RL control methods such as those based on neural networks, and illustrate their relative advantages and drawbacks.
Paper Structure (31 sections, 12 theorems, 108 equations, 9 figures, 2 tables, 2 algorithms)

This paper contains 31 sections, 12 theorems, 108 equations, 9 figures, 2 tables, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Problem Formulation
  3. Carleman Linearization
  4. State-space representation in Carleman space
  5. Derivation of Lyapunov equation
  6. RL Control for Lifted System
  7. Policy iteration for Carleman systems
  8. Theoretical Results
  9. Convergence
  10. $\mathbf{N}^{th}$ order truncation
  11. LQR control objective and policy iteration
  12. Reinforcement learning for the truncated Carleman model
  13. Stability analysis for truncated approximation
  14. Stability analysis for infinite-dimensional system
  15. Online implementation
  16. ...and 16 more sections

Key Result

Theorem 1

Given that $(\mathcal{A},Q^{1/2})$ is controllable, if $K$ is the optimal controller minimizing the objective function eqn:obj for the system eqn:infcarl, then $K$ must satisfy the Lyapunov-like infinite-dimensional equation for a symmetric matrix $P$ given by where $\mathcal{A}_{cl} \psi = \mathcal{A} \psi - B_{\psi} K\psi$.

Figures (9)

  • Figure 1: Flowchart for stability proof of truncated controller
  • Figure 2: Learning of the $2^{nd}$ order RL controller
  • Figure 3: Comparison of learnt controller with HJB solution
  • Figure 4: Learning and control for $\mathbf{N}=2$ Carleman controller
  • Figure 5: Convergence for varying $Q$
  • ...and 4 more figures

Theorems & Definitions (25)

  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • proof
  • Lemma 4
  • proof
  • Theorem 5
  • ...and 15 more