Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On reachability of Markov chains: A long-run average approach

Daniel Avila, Mauricio Junca

Abstract

We consider a Markov control model in discrete time with countable both state space and action space. Using the value function of a suitable long-run average reward problem, we study various reachability/controllability problems. First, we characterize the domain of attraction and escape set of the system, and a generalization called $p$-domain of attraction, using the aforementioned value function. Next, we solve the problem of maximizing the probability of reaching a set $A$ while avoiding a set $B$. Finally, we consider a constrained version of the previous problem where we ask for the probability of reaching the set $B$ to be bounded. In the finite case, we use linear programming formulations to solve these problems. Finally, we apply our results to a example of an object that navigates under stochastic influence.

On reachability of Markov chains: A long-run average approach

Abstract

We consider a Markov control model in discrete time with countable both state space and action space. Using the value function of a suitable long-run average reward problem, we study various reachability/controllability problems. First, we characterize the domain of attraction and escape set of the system, and a generalization called -domain of attraction, using the aforementioned value function. Next, we solve the problem of maximizing the probability of reaching a set while avoiding a set . Finally, we consider a constrained version of the previous problem where we ask for the probability of reaching the set to be bounded. In the finite case, we use linear programming formulations to solve these problems. Finally, we apply our results to a example of an object that navigates under stochastic influence.

Paper Structure

This paper contains 19 sections, 16 theorems, 83 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Markov control model and closed sets
  3. Markov Decision Processes
  4. Long-run average reward problems
  5. Closed sets
  6. Domain of attraction
  7. Reach-avoid problem
  8. Reach with hitting constraint
  9. Linear programming formulations: Finite case
  10. Numerical example
  11. Domain of attraction
  12. Reach and avoid
  13. Reach with constraint
  14. Future work
  15. Proof of Proposition \ref{['HtA']}
  16. ...and 4 more sections

Key Result

Proposition 6

Assume $A$ is a closed set under $\pi\in\Pi$. Then, for any $x\in \mathbb{X}$

Figures (7)

  • Figure 1: Control Matrices $u_1,u_2$
  • Figure 2: Controls
  • Figure 3: $p$-domains $\Lambda_p$ and escape set $\Gamma$.
  • Figure 4: Level sets of $\widetilde{V}(x)$.
  • Figure 5: Trajectories.
  • ...and 2 more figures

Theorems & Definitions (25)

  • Definition 1
  • Definition 2
  • Remark 3
  • Definition 4
  • Remark 5
  • Proposition 6
  • Definition 7
  • Proposition 8
  • Corollary 9
  • Remark 10
  • ...and 15 more