Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness of Online Identification-based Policy Iteration to Noisy Data

Bowen Song, Andrea Iannelli

TL;DR

The paper addresses robustness of indirect data-driven control for unknown discrete-time LTI systems in the LQR setting by integrating online RLS with policy iteration and analyzing the resulting PI-RLS interconnection as an algorithmic dynamical system. It derives finite-sample and ISS-like bounds for RLS under adversarial noise (pointwise and energy-bounded) and establishes convergence guarantees for the coupled ORLS+PI scheme under persistency and stabilization assumptions. The contributions include a dynamical-system reformulation of PI, rigorous convergence/robustness theorems (Theorems 5 and 6), and simulations comparing noise types and against policy-gradient approaches. The results provide practical guidelines for initialization and excitation in noisy environments and highlight the robustness and convergence behavior of concurrent learning and controller design in data-driven LQR problems.

Abstract

This article investigates the core mechanisms of indirect data-driven control for unknown systems, focusing on the application of policy iteration (PI) within the context of the linear quadratic regulator (LQR) optimal control problem. Specifically, we consider a setting where data is collected sequentially from a linear system subject to exogenous process noise, and is then used to refine estimates of the optimal control policy. We integrate recursive least squares (RLS) for online model estimation within a certainty-equivalent framework, and employ PI to iteratively update the control policy. In this work, we investigate first the convergence behavior of RLS under two different models of adversarial noise, namely point-wise and energy bounded noise, and then we provide a closed-loop analysis of the combined model identification and control design process. This iterative scheme is formulated as an algorithmic dynamical system consisting of the feedback interconnection between two algorithms expressed as discrete-time systems. This system theoretic viewpoint on indirect data-driven control allows us to establish convergence guarantees to the optimal controller in the face of uncertainty caused by noisy data. Simulations illustrate the theoretical results.

Robustness of Online Identification-based Policy Iteration to Noisy Data

TL;DR

The paper addresses robustness of indirect data-driven control for unknown discrete-time LTI systems in the LQR setting by integrating online RLS with policy iteration and analyzing the resulting PI-RLS interconnection as an algorithmic dynamical system. It derives finite-sample and ISS-like bounds for RLS under adversarial noise (pointwise and energy-bounded) and establishes convergence guarantees for the coupled ORLS+PI scheme under persistency and stabilization assumptions. The contributions include a dynamical-system reformulation of PI, rigorous convergence/robustness theorems (Theorems 5 and 6), and simulations comparing noise types and against policy-gradient approaches. The results provide practical guidelines for initialization and excitation in noisy environments and highlight the robustness and convergence behavior of concurrent learning and controller design in data-driven LQR problems.

Abstract

This article investigates the core mechanisms of indirect data-driven control for unknown systems, focusing on the application of policy iteration (PI) within the context of the linear quadratic regulator (LQR) optimal control problem. Specifically, we consider a setting where data is collected sequentially from a linear system subject to exogenous process noise, and is then used to refine estimates of the optimal control policy. We integrate recursive least squares (RLS) for online model estimation within a certainty-equivalent framework, and employ PI to iteratively update the control policy. In this work, we investigate first the convergence behavior of RLS under two different models of adversarial noise, namely point-wise and energy bounded noise, and then we provide a closed-loop analysis of the combined model identification and control design process. This iterative scheme is formulated as an algorithmic dynamical system consisting of the feedback interconnection between two algorithms expressed as discrete-time systems. This system theoretic viewpoint on indirect data-driven control allows us to establish convergence guarantees to the optimal controller in the face of uncertainty caused by noisy data. Simulations illustrate the theoretical results.

Paper Structure

This paper contains 18 sections, 13 theorems, 53 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setting and Preliminaries
  3. Policy Iteration
  4. PI System Analysis
  5. Recursive Least Squares
  6. Recursive Least Squares with Adversarial Noise Data
  7. Online Identification-based Policy Iteration
  8. Algorithm Definition
  9. Convergence Analysis of ORLS+PI Algorithm
  10. Simulations
  11. Comparison between different types of noise
  12. Comparison between Policy Iteration and Policy Gradient
  13. Conclusion
  14. Technical Proof
  15. Proof of Theorem \ref{['theoremRLS']}
  16. ...and 3 more sections

Key Result

Theorem 1

Properties of model-based PI 1099755IJRNC If the system dynamics $(A,B)$ are stabilizable, and $K_0$ is stabilizing, then

Figures (4)

  • Figure 1: 2-dimensional Graphic Representation
  • Figure 2: Concurrent identification and policy iteration scheme
  • Figure 3: Comparisons of convergence behaviors of ORLS+PI with different types of noise
  • Figure 4: Comparison of ORLS+PI with ORLS+PG

Theorems & Definitions (24)

  • Theorem 1
  • Theorem 2: Exponential convergence of PI ECC
  • Definition 1
  • Theorem 3
  • Corollary 1
  • Corollary 2: RLS with energy bounded noisy data
  • Remark 1: Remark on $\hat{K}_t$
  • Remark 2: Remark on $e_t$
  • Remark 3: Timestep $t$
  • Theorem 4: ORLS+PI Analysis $1$
  • ...and 14 more