Strongly Polynomial Time Complexity of Policy Iteration for $L_\infty$ Robust MDPs

TL;DR

This work addresses the algorithmic challenge of solving discounted $(s,a)$-rectangular robust MDPs with $\mathbf{L_\infty}$ uncertainty by proving that robust policy iteration can run in strongly polynomial time for fixed $\gamma$. The authors develop a robust policy iteration framework for robust Markov chains (RMC-PI) and extend it to robust MDPs (RMDP-PI), underpinned by a novel potential function and a combinatorial lemma on the sparse structure of policy-improvement transfers. They prove explicit iteration bounds $\mathcal{O}\big(n^4 \log n \cdot \frac{\log((1-\gamma)/n^2)}{\log \gamma}\big)$ for RMC-PI and $\mathcal{O}\big(n m \frac{\log(1-\gamma)}{\log \gamma}\big)$ for RMDP-PI, establishing strong polynomial solvability. The results unify and extend the theory of robust dynamic programming, provide a structural toolset (homotopy-based improvement, potential functions, combinatorial MSB control), and have implications for efficient exact computation in worst-case robust sequential decision-making. This contributes a rigorous, practically relevant foundation for robust planning under uncertainty with fixed discounting.

Abstract

Markov decision processes (MDPs) are a fundamental model in sequential decision making. Robust MDPs (RMDPs) extend this framework by allowing uncertainty in transition probabilities and optimizing against the worst-case realization of that uncertainty. In particular, $(s, a)$-rectangular RMDPs with $L_\infty$ uncertainty sets form a fundamental and expressive model: they subsume classical MDPs and turn-based stochastic games. We consider this model with discounted payoffs. The existence of polynomial and strongly-polynomial time algorithms is a fundamental problem for these optimization models. For MDPs, linear programming yields polynomial-time algorithms for any arbitrary discount factor, and the seminal work of Ye established strongly--polynomial time for a fixed discount factor. The generalization of such results to RMDPs has remained an important open problem. In this work, we show that a robust policy iteration algorithm runs in strongly-polynomial time for $(s, a)$-rectangular $L_\infty$ RMDPs with a constant (fixed) discount factor, resolving an important algorithmic question.

Figures (1)

• Figure 1: Visualization of the Homotopy Algorithm \ref{['algo:homotopy']}.

Theorems & Definitions (53)

• Theorem
• Theorem
• lemma 1
• lemma 2
• definition 1: Potential Function
• lemma 3
• lemma 4
• lemma 5
• proof
• lemma 6