Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving the Model Unavailable MARE using Q-Learning Algorithm

Fei Yan, Jie Gao, Tao Feng, Jianxing Liu

TL;DR

This work tackles the discrete-time modified algebraic Riccati equation ($MARE$) under complete model unavailability. It introduces a novel iterative scheme that recasts the $MARE$ as a sequence of standard DAREs with an input weighting, enabling stabilizing solutions for both single- and multi-input systems and providing clear conditions on $\gamma$ via $\gamma_c$. Building on that, it develops a model-free algorithm based on Q-learning to solve the $MARE$ from input/output data alone, including estimation of the critical value $\gamma_c$. A numerical example demonstrates rapid convergence and high accuracy, indicating practical viability for robust estimation and control in uncertain environments.

Abstract

In this paper, the discrete-time modified algebraic Riccati equation (MARE) is solved when the system model is completely unavailable. To achieve this, firstly a brand new iterative method based on the standard discrete-time algebraic Riccati equation (DARE) and its input weighting matrix is proposed to solve the MARE. For the single-input case, the iteration can be initialized by an arbitrary positive input weighting if and only if the MARE has a stabilizing solution; nevertheless a pre-given input weighting matrix of a sufficiently large magnitude is used to perform the iteration for the multi-input case when the characteristic parameter belongs to a specified subset. Benefit from the developed specific iteration structure, the Q-learning (QL) algorithm can be employed to subtly solve the MARE where only the system input/output data is used thus the system model is not required. Finally, a numerical simulation example is given to verify the effectiveness of the theoretical results and the algorithm.

Solving the Model Unavailable MARE using Q-Learning Algorithm

TL;DR

This work tackles the discrete-time modified algebraic Riccati equation () under complete model unavailability. It introduces a novel iterative scheme that recasts the as a sequence of standard DAREs with an input weighting, enabling stabilizing solutions for both single- and multi-input systems and providing clear conditions on via . Building on that, it develops a model-free algorithm based on Q-learning to solve the from input/output data alone, including estimation of the critical value . A numerical example demonstrates rapid convergence and high accuracy, indicating practical viability for robust estimation and control in uncertain environments.

Abstract

In this paper, the discrete-time modified algebraic Riccati equation (MARE) is solved when the system model is completely unavailable. To achieve this, firstly a brand new iterative method based on the standard discrete-time algebraic Riccati equation (DARE) and its input weighting matrix is proposed to solve the MARE. For the single-input case, the iteration can be initialized by an arbitrary positive input weighting if and only if the MARE has a stabilizing solution; nevertheless a pre-given input weighting matrix of a sufficiently large magnitude is used to perform the iteration for the multi-input case when the characteristic parameter belongs to a specified subset. Benefit from the developed specific iteration structure, the Q-learning (QL) algorithm can be employed to subtly solve the MARE where only the system input/output data is used thus the system model is not required. Finally, a numerical simulation example is given to verify the effectiveness of the theoretical results and the algorithm.
Paper Structure (8 sections, 3 theorems, 38 equations, 4 figures)

This paper contains 8 sections, 3 theorems, 38 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Solving MARE by a new iterative method
  4. Single-Input Case
  5. Multi-Input Case
  6. Solving MARE without system model
  7. Simulation
  8. Conclusion

Key Result

Lemma 1

Consider the following DARE where $(A,B)$ is stabilizable and $(A,\sqrt{Q})$ is detectable. Defining that $\beta = {b^T}Xb/\sigma$, then the scalar $\beta$ converges to its minimum as $\sigma \to +\infty$ and the infimum is given by

Figures (4)

  • Figure 1: Convergence process of $\gamma_c$
  • Figure 2: Error between matrix $\mathbb{H}^{t}$ and the true value of $\mathbb{H}$
  • Figure 3: The norm of ${\omega_n} - \frac{{1 - \gamma }}{\gamma }{\mathbb{H}_{uu}} - r$
  • Figure 4: The norm of $X_n-\Delta_n (X_n)$

Theorems & Definitions (6)

  • Lemma 1
  • Theorem 1
  • Remark 1
  • Theorem 2
  • Remark 2
  • Remark 3