Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memoryless Strategies in Stochastic Reachability Games

Stefan Kiefer, Richard Mayr, Mahsa Shirmohammadi, Patrick Totzke

TL;DR

The paper investigates memoryless strategies in two-player concurrent stochastic reachability games, focusing on when Max can achieve optimal or near-optimal reachability probabilities. It introduces a martingale-based framework and a leaky-safety reduction to convert reachability questions into safety problems, enabling constructive existence results. The authors provide an alternative, simpler proof that Max, when optimal strategies exist, can be chosen as randomized memoryless in finite-state, finite-Min-action games, and extend this to countably infinite Max actions; they also establish ε-optimal and optimal results for reachability under finite action assumptions, and present counterexamples illustrating the necessity of those finiteness conditions. The work clarifies the boundary between memoryless sufficiency and the need for memory in more general (infinite) settings, with implications for algorithmic design in stochastic game analysis. Overall, it offers a principled, construction-based approach to memoryless strategies in concurrency-laden reachability games and delineates the role of finiteness in guaranteeing optimality.

Abstract

We study concurrent stochastic reachability games played on finite graphs. Two players, Max and Min, seek respectively to maximize and minimize the probability of reaching a set of target states. We prove that Max has a memoryless strategy that is optimal from all states that have an optimal strategy. Our construction provides an alternative proof of this result by Bordais, Bouyer and Le Roux, and strengthens it, as we allow Max's action sets to be countably infinite.

Memoryless Strategies in Stochastic Reachability Games

TL;DR

The paper investigates memoryless strategies in two-player concurrent stochastic reachability games, focusing on when Max can achieve optimal or near-optimal reachability probabilities. It introduces a martingale-based framework and a leaky-safety reduction to convert reachability questions into safety problems, enabling constructive existence results. The authors provide an alternative, simpler proof that Max, when optimal strategies exist, can be chosen as randomized memoryless in finite-state, finite-Min-action games, and extend this to countably infinite Max actions; they also establish ε-optimal and optimal results for reachability under finite action assumptions, and present counterexamples illustrating the necessity of those finiteness conditions. The work clarifies the boundary between memoryless sufficiency and the need for memory in more general (infinite) settings, with implications for algorithmic design in stochastic game analysis. Overall, it offers a principled, construction-based approach to memoryless strategies in concurrency-laden reachability games and delineates the role of finiteness in guaranteeing optimality.

Abstract

We study concurrent stochastic reachability games played on finite graphs. Two players, Max and Min, seek respectively to maximize and minimize the probability of reaching a set of target states. We prove that Max has a memoryless strategy that is optimal from all states that have an optimal strategy. Our construction provides an alternative proof of this result by Bordais, Bouyer and Le Roux, and strengthens it, as we allow Max's action sets to be countably infinite.
Paper Structure (9 sections, 8 theorems, 33 equations, 4 figures)

This paper contains 9 sections, 8 theorems, 33 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Background.
  3. Our contribution.
  4. Preliminaries
  5. Martingales for Safety
  6. Optimal Safety
  7. $\varepsilon$-Optimal Reachability
  8. Optimal Reachability
  9. Conclusion and Related Work

Key Result

lemma 1

Let $\sigma$ and $\pi$ be Max and Min strategies, respectively. Suppose $v : S \to [0,1]$ is a function with $v(\bot) = 0$ such that $\langle p(s_h,\sigma(h),\pi(h)),v\rangle \ge v(s_h)$ holds for all histories $h$. Then ${\mathcal{P}}_{s_0,\sigma,\pi}(\mathtt{Avoid}(\bot)) \ge v(s_0)$ holds for all

Figures (4)

  • Figure 1: A finite-state MDP where Max has no $\varepsilon$-optimal memoryless strategy for safety objective $\mathtt{Avoid}(\bot)$. The state $\bot$ is a sink state. The set $\{a_i \mid i\in \mathbb{N}\}$ of Max's actions at $s$ is countable, while Min only has a single action $b$.
  • Figure 2: The snowball game where Max has no optimal strategy for $\mathtt{Reach}(\top)$. The states $\top$ and $\bot$ are sink states. Max's action set at $s$ is $\{hide,run\}$, shown as $h$ and $r$ in the figure, while Min's action set is $\{wait,throw\}$, shown as $w$ and $t$.
  • Figure 3: A game where Max has no memoryless optimal strategy for $\mathtt{Reach}(\top)$. The states $\top$ and $\bot$ are sink states. Min's action set at $s_0$ is $\{b_q \mid q \in (\frac{1}{2},1) \cap \mathbb{Q}\}$, while Max has a single action $a$ at $s_0$. The game in dashed box is the snowball game from \ref{['fig:hiderun']}.
  • Figure 4: A finitely-branching turn-based reachability game ${\mathcal{G}}$ with initial state $u_1$, where optimal Max strategies cannot be memoryless (and cannot even be Markov). For clarity, we have drawn several copies of the target state $t$. The number $y_i$ is defined as $\frac{1}{2} - \frac{1}{2^{i+1}}$.

Theorems & Definitions (16)

  • lemma 1
  • proof
  • proposition 1
  • proof
  • proposition 2
  • remark 1
  • lemma 2
  • proof
  • lemma 3
  • proof
  • ...and 6 more