Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds

Yahya Sattar, Zhe Du, Davoud Ataee Tarzanagh, Laura Balzano, Necmiye Ozay, Samet Oymak

TL;DR

It is proved that when the episode lengths are appropriately chosen, the proposed adaptive control scheme achieves $\mathcal{O}(\sqrt{T})$ regret, which can be improved to $\mathcal{O}(polylog(T))$ with partial knowledge of the system.

Abstract

Learning how to effectively control unknown dynamical systems is crucial for intelligent autonomous systems. This task becomes a significant challenge when the underlying dynamics are changing with time. Motivated by this challenge, this paper considers the problem of controlling an unknown Markov jump linear system (MJS) to optimize a quadratic objective. By taking a model-based perspective, we consider identification-based adaptive control of MJSs. We first provide a system identification algorithm for MJS to learn the dynamics in each mode as well as the Markov transition matrix, underlying the evolution of the mode switches, from a single trajectory of the system states, inputs, and modes. Through martingale-based arguments, sample complexity of this algorithm is shown to be $\mathcal{O}(1/\sqrt{T})$. We then propose an adaptive control scheme that performs system identification together with certainty equivalent control to adapt the controllers in an episodic fashion. Combining our sample complexity results with recent perturbation results for certainty equivalent control, we prove that when the episode lengths are appropriately chosen, the proposed adaptive control scheme achieves $\mathcal{O}(\sqrt{T})$ regret, which can be improved to $\mathcal{O}(polylog(T))$ with partial knowledge of the system. Our proof strategy introduces innovations to handle Markovian jumps and a weaker notion of stability common in MJSs. Our analysis provides insights into system theoretic quantities that affect learning accuracy and control performance. Numerical simulations are presented to further reinforce these insights.

Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds

TL;DR

It is proved that when the episode lengths are appropriately chosen, the proposed adaptive control scheme achieves regret, which can be improved to with partial knowledge of the system.

Abstract

Learning how to effectively control unknown dynamical systems is crucial for intelligent autonomous systems. This task becomes a significant challenge when the underlying dynamics are changing with time. Motivated by this challenge, this paper considers the problem of controlling an unknown Markov jump linear system (MJS) to optimize a quadratic objective. By taking a model-based perspective, we consider identification-based adaptive control of MJSs. We first provide a system identification algorithm for MJS to learn the dynamics in each mode as well as the Markov transition matrix, underlying the evolution of the mode switches, from a single trajectory of the system states, inputs, and modes. Through martingale-based arguments, sample complexity of this algorithm is shown to be . We then propose an adaptive control scheme that performs system identification together with certainty equivalent control to adapt the controllers in an episodic fashion. Combining our sample complexity results with recent perturbation results for certainty equivalent control, we prove that when the episode lengths are appropriately chosen, the proposed adaptive control scheme achieves regret, which can be improved to with partial knowledge of the system. Our proof strategy introduces innovations to handle Markovian jumps and a weaker notion of stability common in MJSs. Our analysis provides insights into system theoretic quantities that affect learning accuracy and control performance. Numerical simulations are presented to further reinforce these insights.

Paper Structure

This paper contains 37 sections, 31 theorems, 170 equations, 3 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries and Problem Setup
  4. Markov Jump Linear Systems
  5. System Identification
  6. Adaptive Quadratic Control
  7. System Identification for MJS
  8. Adaptive Control for MJS-LQR
  9. Regret Analysis
  10. Two Special Cases
  11. Regret under uniform stability
  12. Partial knowledge of dynamics
  13. Discussion
  14. Initial Stabilizing Controllers
  15. Offline Data-Driven Control
  16. ...and 22 more sections

Key Result

Theorem 1

Suppose we run Algorithm Alg_MJS-SYSID with the trajectory length $T \geq \max \{2 T_0, \hat{\mathcal{O}}( \frac{(n+p)\log(T)}{\pi_{\min}(1 - \varrho)})\}$, where $T_0 := t_{\rm MC}(\pi_{\min}/2)$ and $\varrho:= \hat{\mathcal{O}}(\frac{1}{\pi_{\min}}\sqrt{\frac{\pi_{\max}T_0}{T})}$. Suppose, $\{{\bo

Figures (3)

  • Figure 1: State trajectories for a two-modes MJS: Mode 1: $x_{t+1} = 1.2 x_{t}$, Mode 2: $x_{t+1} = 0.7 x_{t}$, Markov matrix $[[0.6, 0.4]^\top, [0.3, 0.7]^\top]^\top$, and $x_0=1$. Blue and red curves: mode switching sequences $\Omega_1=\{ 1,1,\dots \}$ and $\Omega_2=\{ 2,2,\dots \}$. Yellow curve: average over all realizations. Gray area: region for all possible trajectories.
  • Figure 2: Performance profiles of MJS-SYSID with varying: (a) process noise $\sigma_{\boldsymbol{w}}$, (b) exploration noise $\sigma_{\boldsymbol{z}}$, (c) state dimension $n$, and (d) number of modes $s$.
  • Figure 3: Performance profiles of Adaptive MJS-LQR with varying: (a) process noise $\sigma_{\boldsymbol{w}}$, (b) number of modes $s$, (c) state dimension $n$.

Theorems & Definitions (60)

  • Definition 1: Mean-square stability costa2006discrete
  • Definition 2: Markov chain mixing
  • Theorem 1: Identification of MJS
  • Corollary 1: Identification with known ${\boldsymbol{B}}_{1:s}$
  • Theorem 2: Sub-linear regret
  • Theorem 3: Regret under uniform stability
  • Theorem 4: Random regret
  • Corollary 2: Poly-logarithmic regret
  • Definition 3
  • Definition 4
  • ...and 50 more