Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Direct Data-Driven Discounted Infinite Horizon Linear Quadratic Regulator with Robustness Guarantees

Ramin Esmzad, Hamidreza Modares

TL;DR

An adaptive dynamic programming formalization of the LQR is presented that relies on solving a Bellman inequality and it is shown that this system characterization leads to a closed-loop system with multiplicative and additive noise, enabling the application of distributional robust control techniques.

Abstract

This paper presents a one-shot learning approach with performance and robustness guarantees for the linear quadratic regulator (LQR) control of stochastic linear systems. Even though data-based LQR control has been widely considered, existing results suffer either from data hungriness due to the inherently iterative nature of the optimization formulation (e.g., value learning or policy gradient reinforcement learning algorithms) or from a lack of robustness guarantees in one-shot non-iterative algorithms. To avoid data hungriness while ensuing robustness guarantees, an adaptive dynamic programming formalization of the LQR is presented that relies on solving a Bellman inequality. The control gain and the value function are directly learned by using a control-oriented approach that characterizes the closed-loop system using data and a decision variable from which the control is obtained. This closed-loop characterization is noise-dependent. The effect of the closed-loop system noise on the Bellman inequality is considered to ensure both robust stability and suboptimal performance despite ignoring the measurement noise. To ensure robust stability, it is shown that this system characterization leads to a closed-loop system with multiplicative and additive noise, enabling the application of distributional robust control techniques. The analysis of the suboptimality gap reveals that robustness can be achieved without the need for regularization or parameter tuning. The simulation results on the active car suspension problem demonstrate the superiority of the proposed method in terms of robustness and performance gap compared to existing methods.

Direct Data-Driven Discounted Infinite Horizon Linear Quadratic Regulator with Robustness Guarantees

TL;DR

An adaptive dynamic programming formalization of the LQR is presented that relies on solving a Bellman inequality and it is shown that this system characterization leads to a closed-loop system with multiplicative and additive noise, enabling the application of distributional robust control techniques.

Abstract

This paper presents a one-shot learning approach with performance and robustness guarantees for the linear quadratic regulator (LQR) control of stochastic linear systems. Even though data-based LQR control has been widely considered, existing results suffer either from data hungriness due to the inherently iterative nature of the optimization formulation (e.g., value learning or policy gradient reinforcement learning algorithms) or from a lack of robustness guarantees in one-shot non-iterative algorithms. To avoid data hungriness while ensuing robustness guarantees, an adaptive dynamic programming formalization of the LQR is presented that relies on solving a Bellman inequality. The control gain and the value function are directly learned by using a control-oriented approach that characterizes the closed-loop system using data and a decision variable from which the control is obtained. This closed-loop characterization is noise-dependent. The effect of the closed-loop system noise on the Bellman inequality is considered to ensure both robust stability and suboptimal performance despite ignoring the measurement noise. To ensure robust stability, it is shown that this system characterization leads to a closed-loop system with multiplicative and additive noise, enabling the application of distributional robust control techniques. The analysis of the suboptimality gap reveals that robustness can be achieved without the need for regularization or parameter tuning. The simulation results on the active car suspension problem demonstrate the superiority of the proposed method in terms of robustness and performance gap compared to existing methods.
Paper Structure (11 sections, 8 theorems, 84 equations, 2 tables)

This paper contains 11 sections, 8 theorems, 84 equations, 2 tables.

Table of Contents

  1. Introduction
  2. Model-Based LQR Formulation
  3. Problem Statement
  4. LMI Formulation of Model-Based LQR Problem
  5. Data-based LQR Formulation
  6. Indirect data-driven (CE approach)
  7. Direct data-driven
  8. MSS Direct Data-Driven LQR
  9. Suboptimality gap
  10. Monte Carlo Simulations
  11. Conclusion

Key Result

Lemma 1

Let $A \in \mathbb{R}^{n \times n}$. Then, for $\gamma > 1 - \frac{\lambda_{\min}(M)}{\lambda_{\max}(A^\top P A)}$, the spectral radius $\rho(A) < 1$ if and only if, for each given positive definite matrix $M \in \mathbb{R}^{n \times n}$, there exists a unique positive definite matrix $P \in \mathbb

Theorems & Definitions (24)

  • Definition 1
  • Remark 1
  • Lemma 1
  • Remark 2
  • Lemma 2
  • Lemma 3
  • Remark 3
  • Remark 4
  • Remark 5
  • Lemma 4
  • ...and 14 more