Computing Lower and Upper Hitting Probabilities for Imprecise Markov Chains

TL;DR

The paper tackles the problem of computing lower and upper hitting probabilities in imprecise Markov chains modeled by credal sets, addressing how reachability should be defined under uncertainty. It establishes fixed-point characterizations for nontrivial states and develops iterative algorithms that alternate solving linear systems with extreme-point updates, backed by convergence proofs and practical starting-point strategies. Empirical results show the proposed methods converge rapidly in practice, often far below worst-case iteration bounds, highlighting their suitability for robust reachability analysis under transition uncertainty. These contributions provide scalable, conservative tools for planning and control under imprecision in stochastic dynamics.

Abstract

We study the computation of lower and upper probabilities of hitting a target set of states for imprecise Markov chains. For these, transition uncertainty is modelled by a convex set of transition matrices. In the precise case, hitting probabilities are the minimal nonnegative solution of a linear system and admit a closed-form expression. We study the notion of reachability in the imprecise setting. The literature review highlights few different definitions of lower reachability; thus we explore the relations among them, presenting examples to clarify their logical implications. Using this revised definition of reachability for imprecise Markov chain, we partition the state space into classes of states whose hitting probabilities are trivially zero or one and those which require further computation. For these nontrivial states, we show that lower and upper hitting probabilities are the unique solutions of two nonlinear fixed-point equations. For the practical computation of lower and upper hitting probabilities, we propose iterative algorithms that alternate between solving a linear system and choosing an extreme point from the set of transition matrices. Numerical experiments demonstrate that, in practice, these algorithms converge in substantially fewer iterations than the theoretically established worst-case bound.

Figures (4)

• Figure 1: Credal set of the first row of $\mathcal{T}(6)$.
• Figure 2: The red and blue lines represent the average number of iterations needed to compute lower and upper hitting probabilities, respectively, as a function of the average degree of each node of the graph, $\lambda$. The two magenta shaded regions correspond to the intervals within one standard deviation ($\pm \sigma$) and approximately $95\%$ confidence ($\pm 1.96\sigma$) around the mean number of iterations.
• Figure 3: Number of instances, out of 1000, on which the algorithm for upper hitting probabilities converged in a given number of iterations, for different values of the average degree $\lambda$. The graph has $N=50$ nodes and the set of transition matrices is generated by taking the convex hull of transition matrices sampled uniformly at random.
• Figure 4: Average number of iterations as a function of the number of states $N$ and the average degree $\lambda$. Each red dot on the blue line marks, for a fixed number of states $N$, the average degree $\lambda^*$ that attains the maximum average number of iterations.

Theorems & Definitions (60)

• Definition 2.1: Reachability norrisbook1997markov
• Proposition 2.2: norrisbook1997markov
• Definition 2.3
• Definition 2.4
• Proposition 2.5: norrisbook1997markov
• Lemma 2.6
• Lemma 2.7
• Lemma 2.8
• Lemma 2.9
• Corollary 2.10