Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Resilient Strategies for Stochastic Systems: How Much Does It Take to Break a Winning Strategy?

Kush Grover, Markel Zubia, Debraj Chakraborty, Muqsit Azeem, Nils Jansen, Jan Kretinsky

TL;DR

This work introduces the concept of resilience in the stochastic setting and presents a comprehensive set of fundamental problems for Markov decision processes with reachability and safety objectives, which also smoothly extend to stochastic games.

Abstract

We study the problem of resilient strategies in the presence of uncertainty. Resilient strategies enable an agent to make decisions that are robust against disturbances. In particular, we are interested in those disturbances that are able to flip a decision made by the agent. Such a disturbance may, for instance, occur when the intended action of the agent cannot be executed due to a malfunction of an actuator in the environment. In this work, we introduce the concept of resilience in the stochastic setting and present a comprehensive set of fundamental problems. Specifically, we discuss such problems for Markov decision processes with reachability and safety objectives, which also smoothly extend to stochastic games. To account for the stochastic setting, we provide various ways of aggregating the amounts of disturbances that may have occurred, for instance, in expectation or in the worst case. Moreover, to reason about infinite disturbances, we use quantitative measures, like their frequency of occurrence.

Resilient Strategies for Stochastic Systems: How Much Does It Take to Break a Winning Strategy?

TL;DR

This work introduces the concept of resilience in the stochastic setting and presents a comprehensive set of fundamental problems for Markov decision processes with reachability and safety objectives, which also smoothly extend to stochastic games.

Abstract

We study the problem of resilient strategies in the presence of uncertainty. Resilient strategies enable an agent to make decisions that are robust against disturbances. In particular, we are interested in those disturbances that are able to flip a decision made by the agent. Such a disturbance may, for instance, occur when the intended action of the agent cannot be executed due to a malfunction of an actuator in the environment. In this work, we introduce the concept of resilience in the stochastic setting and present a comprehensive set of fundamental problems. Specifically, we discuss such problems for Markov decision processes with reachability and safety objectives, which also smoothly extend to stochastic games. To account for the stochastic setting, we provide various ways of aggregating the amounts of disturbances that may have occurred, for instance, in expectation or in the worst case. Moreover, to reason about infinite disturbances, we use quantitative measures, like their frequency of occurrence.
Paper Structure (43 sections, 16 theorems, 12 equations, 5 figures, 1 table)

This paper contains 43 sections, 16 theorems, 12 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction and Motivation
  2. Resilient Strategies.
  3. Resilience in Stochastic Systems.
  4. Assumptions on Model Availability.
  5. Disturbances with Different Costs.
  6. Summary of Our Contribution:
  7. Related Work.
  8. Preliminaries
  9. Stochastic Games
  10. Paths or Runs
  11. Strategies
  12. Markov Decision Processes (MDP) and Markov Chains (MC)
  13. End Component (EC)
  14. Rewards or Costs.
  15. Specifications.
  16. ...and 28 more sections

Key Result

Lemma 1

For a memoryless Player 1 strategy $\pi$, there exist a bijective function $h$ that maps pairs $(\sigma, \delta)$ of Player 2 and disturbance strategies in $\mathcal{G}$ to a strategy $\mu$ in $M_\pi$ such that

Figures (5)

  • Figure 1: Drone navigation task with wind disturbances.
  • Figure 2: For this SGD, if the objective is to reach $G$ with probability $> 0.4$, then the worst-case breaking point is 2, as disturbing just once will not break the policy, and the expected breaking point is 1.1 via an adversary that always disturbs in $s_0$ and with a probability of 0.2 in $s_1$.
  • Figure 3: A gadget in the unfolded stochastic game.
  • Figure 4: Gadget for converting an SGD into an SG for the Expected Breaking Point.
  • Figure 5: An SGD where the most resilient $\pi$ must have memory even if $\delta$ is memoryless (left), and one where the optimal $\delta$ must rely on memory even if $\pi$ is memoryless (right).

Theorems & Definitions (25)

  • Definition 1
  • Definition 2
  • Remark 1
  • Lemma 1
  • Lemma 2
  • Definition 3
  • Lemma 3
  • Theorem 1
  • Lemma 4
  • Theorem 2
  • ...and 15 more