Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stability of multiplexed NCS based on an epsilon-greedy algorithm for communication selection

Harsh Oza, Irinel-Constantin Morarescu, Vineeth S. Varma, Ravi Banavar

TL;DR

This letter proposes an epsilon-greedy algorithm for the selection of the communication sequence that also ensures Mean Square Stability (MSS) and proposes to use the epsilon-greedy algorithm with the epsilon that satisfies MSS conditions for training a Deep Q Network (DQN).

Abstract

In this letter, we study a Networked Control System (NCS) with multiplexed communication and Bernoulli packet drops. Multiplexed communication refers to the constraint that transmission of a control signal and an observation signal cannot occur simultaneously due to the limited bandwidth. First, we propose an epsilon-greedy algorithm for the selection of the communication sequence that also ensures Mean Square Stability (MSS). We formulate the system as a Markovian Jump Linear System (MJLS) and provide the necessary conditions for MSS in terms of Linear Matrix Inequalities (LMIs) that need to be satisfied for three corner cases. We prove that the system is MSS for any convex combination of these three corner cases. Furthermore, we propose to use the epsilon-greedy algorithm with the epsilon that satisfies MSS conditions for training a Deep Q Network (DQN). The DQN is used to obtain an optimal communication sequence that minimizes a quadratic cost. We validate our approach with a numerical example that shows the efficacy of our method in comparison to the round-robin and a random scheme.

Stability of multiplexed NCS based on an epsilon-greedy algorithm for communication selection

TL;DR

This letter proposes an epsilon-greedy algorithm for the selection of the communication sequence that also ensures Mean Square Stability (MSS) and proposes to use the epsilon-greedy algorithm with the epsilon that satisfies MSS conditions for training a Deep Q Network (DQN).

Abstract

In this letter, we study a Networked Control System (NCS) with multiplexed communication and Bernoulli packet drops. Multiplexed communication refers to the constraint that transmission of a control signal and an observation signal cannot occur simultaneously due to the limited bandwidth. First, we propose an epsilon-greedy algorithm for the selection of the communication sequence that also ensures Mean Square Stability (MSS). We formulate the system as a Markovian Jump Linear System (MJLS) and provide the necessary conditions for MSS in terms of Linear Matrix Inequalities (LMIs) that need to be satisfied for three corner cases. We prove that the system is MSS for any convex combination of these three corner cases. Furthermore, we propose to use the epsilon-greedy algorithm with the epsilon that satisfies MSS conditions for training a Deep Q Network (DQN). The DQN is used to obtain an optimal communication sequence that minimizes a quadratic cost. We validate our approach with a numerical example that shows the efficacy of our method in comparison to the round-robin and a random scheme.
Paper Structure (16 sections, 1 theorem, 31 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 16 sections, 1 theorem, 31 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Plant and Controller Model
  4. Networked System Model
  5. Switching Strategy
  6. $\varepsilon$- Greedy Algorithm for Switching
  7. Switching Probabilities with $\varepsilon$- Greedy Algorithm
  8. Formulation as a Markovian Jump Linear System
  9. Mean Square Stability of MJLS
  10. Corner Cases
  11. Mode Transition Probabilities for Corner Cases
  12. Optimal Switching Strategy
  13. $\mathsf{Q}$-Learning
  14. Deep $\mathsf{Q}$-Learning
  15. Numerical Experiments
  16. ...and 1 more sections

Key Result

Theorem 1

Given a $\delta \in \lcrc{0}{1}$, with mode transition probability matrix $\mathsf{P}^{(c)}$ the origin of the system eq: overall system is MSS if there exist an $\Bar{\varepsilon} \in \loro{0}{1}$ and a symmetric positive definite matrix $V$, such that the following holds for all $c \in \{1,2,3\}$. Furthermore, with this $\varepsilon$, the origin of the system eq: overall system is MSS under the

Figures (7)

  • Figure 1: Schematic of a Networked Control System with information multiplexing and packet drops in the network
  • Figure 2: Markov Jump Linear System with associated mode transition probabilities for a general case.
  • Figure 3: $\Bar{\varepsilon}$ that satisfies MSS conditions for all three corner cases for different values of $\delta$
  • Figure 4: Comparison of average reward ($\lambda=0.5$) of DQN method with round robin and random switching scheme
  • Figure 5: Low success probability, with high $\varepsilon$
  • ...and 2 more figures

Theorems & Definitions (5)

  • Remark 1
  • Definition 1: Mode Transition Probability
  • Definition 2: Mean Square Stability
  • Theorem 1
  • proof