Howard's Policy Iteration is Subexponential for Deterministic Markov Decision Problems with Rewards of Fixed Bit-size and Arbitrary Discount Factor

TL;DR

This paper addresses whether Howard's Policy Iteration (HPI) can be shown to run in subexponential time on deterministic MDPs when rewards have bounded bit-size and the discount factor is arbitrary. The authors introduce a threshold discount factor $\gamma_{Q}$ and a Q-difference sign polynomial $f^{\pi}_{s,a,a'}(\gamma)$ to capture the sign of Q-value improvements across states, and they bound the degree and height of these polynomials. A root-separation analysis for integer-coefficient polynomials then yields a bound on $\gamma_{Q}$, which in turn bounds the number of iterations HPI can take, yielding $nk \cdot u \exp(u)$ with $u = O\left(\sqrt{nb} \log\frac{n}{b} + b\right)$, i.e., subexponential in the number of states $n$ when $b$ is constant. The results extend to the two-valued reward case and provide a novel connection between polynomial root bounds and algorithmic running time, offering practical insights for deterministic MDPs with practical reward encodings.

Abstract

Howard's Policy Iteration (HPI) is a classic algorithm for solving Markov Decision Problems (MDPs). HPI uses a "greedy" switching rule to update from any non-optimal policy to a dominating one, iterating until an optimal policy is found. Despite its introduction over 60 years ago, the best-known upper bounds on HPI's running time remain exponential in the number of states -- indeed even on the restricted class of MDPs with only deterministic transitions (DMDPs). Meanwhile, the tightest lower bound for HPI for MDPs with a constant number of actions per state is only linear. In this paper, we report a significant improvement: a subexponential upper bound for HPI on DMDPs, which is parameterised by the bit-size of the rewards, while independent of the discount factor. The same upper bound also applies to DMDPs with only two possible rewards (which may be of arbitrary size).

Figures (2)

• Figure 1: In (a) we see graph $G_{M}$ induced by DMDP $M$ with $S = \{0, 1, 2, 3\}$, $A = \{0 \text{ (solid)}, 1 \text{ (dashed)}, 2 \text{ (dotted)}\}$. Each transition is annotated with the reward. Subfigure (b) shows the subgraph $G_{M, \pi}$ obtained by fixing policy $\pi$ such that $\pi(0) = 0, \pi(1) = 1, \pi(2) = 0, \pi(3) = 2$. Starting from each state $s$ in $G_{M, \pi}$ is a (possibly null) path $P^{\pi}_{s}$ and a cycle $C^{\pi}_{s}$ (defined in Section \ref{['sec:path-cycle']} as a sequence of state-action pairs). For example: $P^{\pi}_{1} = \emptyset, C^{\pi}_{1} = \langle (1, 1) \rangle$; $P^{\pi}_{2} = \emptyset, C^{\pi}_{2} = \langle (2, 0),(0,0) \rangle$; and $P^{\pi}_{3} = \langle (3, 2) \rangle, C^{\pi}_{3} = \langle (2, 0),(0,0) \rangle$. In (c) we see $G_{M_{m}}$ for $m = 2$, containing 6 states and 2 actions. The family $M_{m}$ is described in Section \ref{['sec:Avg-vs-Disc']}. Subfigure (d) shows the subgraph of $G_{M}$ induced by some fixed policy $\pi_{1}$. Notice that state $0$ is on a cycle with $4$ edges, exactly $2$ of which have a reward of $1$.
• Figure 2: Induced graph of a DMDP with states $S = \{0, 1, \dots, n - 1\}$ and actions $0$ (dashed) and $1$ (solid). Annotations show the reward on each transition. Discount factor $\gamma$ is any element of $(0, 1)$. Policies are encoded as $n$-bit strings: bit $i$ specifies the action taken at state $i - 1$. For starting policy $0^{n}$, the only improving action is at state $n - 1$. Hence every PI algorithm must proceed from $0^{n}$ to $0^{n - 1}1$. Now, for policy $0^{n - 1}1$, the only improving action is at state $n - 2$. Hence every PI algorithm must proceed from $0^{n - 1}1$ to $0^{n - 2}1^{2}$. The same pattern continues until, after $n$ total iterations, the optimal policy $1^{n}$ is reached.

Theorems & Definitions (10)

• Proposition 1.1
• Theorem 1.2
• Theorem 2.1: Policy Improvement
• Theorem 3.1
• Lemma 4.1
• proof
• Theorem 4.2
• Proposition 4.3
• Proposition 4.4
• Theorem 4.5