Robust Deterministic Policies for Markov Decision Processes under Budgeted Uncertainty

TL;DR

This paper studies the computation of robust deterministic policies for Markov Decision Processes (MDPs) in the LDST model and derives strong inapproximability results for computing robust deterministic policies as well as $\Sigma_2^p-hardness, indicating that the general problem does not even admit a compact mixed integer programming formulation.

Abstract

This paper studies the computation of robust deterministic policies for Markov Decision Processes (MDPs) in the Lightning Does Not Strike Twice (LDST) model of Mannor, Mebel and Xu (ICML '12). In this model, designed to provide robustness in the face of uncertain input data while not being overly conservative, transition probabilities and rewards are uncertain and the uncertainty set is constrained by a budget that limits the number of states whose parameters can deviate from their nominal values. Mannor et al. (ICML '12) showed that optimal randomized policies for MDPs in the LDST regime can be efficiently computed when only the rewards are affected by uncertainty. In contrast to these findings, we observe that the computation of optimal deterministic policies is $N\!P$-hard even when only a single terminal reward may deviate from its nominal value and the MDP consists of $2$ time periods. For this hard special case, we then derive a constant-factor approximation algorithm by combining two relaxations based on the Knapsack Cover and Generalized Assignment problem, respectively. For the general problem with possibly a large number of deviations and a longer time horizon, we derive strong inapproximability results for computing robust deterministic policies as well as $Σ_2^p$-hardness, indicating that the general problem does not even admit a compact mixed integer programming formulation.

Figures (5)

• Figure 2.1: Example showing that, in the LDST model, the reward of a randomized policy can exceed that of an optimal deterministic policy. Each round node represents a state, each square represent an action. Numbers on the arcs denote the (nominal) transition probabilities. Numbers next to the terminal states indicate nominal (first number, black) and worst-case (second number, red) reward, respectively.
• Figure 4.1: Construction of the reduction from 3-Partition to $2$-Stage Reward-DLP.
• Figure 4.2: The construction of the reduction from $3$-Vertex-Disjoint Paths problem in $D$ to Reward-DLP is shown in this figure, where the actions in each state correspond to the edges that leave from the respective node in $D$ and $\delta=\varepsilon+\varepsilon^2+\varepsilon^3$. Note that in the example, no pair of vertex-disjoint $s_1$-$t_1$- and $s_3$-$t_3$-paths exists in $D$ and similarly, as any such paths would intersect at node $v$. Similarly, no policy can reach both terminal states $t_1$ and $t_3$ with positive probability, as only one of the actions can be chosen in state $v$, deterministically leading to either $t_1$ or $t_3$ from that state.
• Figure 6.1: Construction of the Reduction from max-min VC to Transition-DLP. The solid gray arrows represent an nominal deterministic outcomes of a transition, with the dashed gray line representing the alternative transition outcomes from the uncertainty set.
• Figure 6.2: Construction for the reduction from 3-SAT to Transition-DLP. The solid gray arrows represent an nominal deterministic outcomes of a transition, with the dashed gray line representing the alternative transition outcomes from the uncertainty set. There are at most $k = 2$ actions for which the transition can deviate from its nominal state.

Theorems & Definitions (32)

• theorem 1
• theorem 2
• theorem 3
• theorem 4
• theorem 5
• remark 1
• theorem 5
• proof
• theorem 5
• proof