Solving Robust Markov Decision Processes: Generic, Reliable, Efficient
Tobias Meggendorfer, Maximilian Weininger, Patrick Wienhöft
TL;DR
This work extends robust Markov decision processes (RMDPs) to handle arbitrary, and even non-polytopic, uncertainty sets while supporting multiple objectives including undiscounted total reward, long-run average reward, and stochastic shortest path. It builds a generic, reliable, and efficient framework by reducing RMDPs to turn-based stochastic games and solving them implicitly, thereby avoiding explicit construction of potentially infinite SGs and providing precision guarantees at any computation stage. The key innovations are the implicit robust value iteration with a guaranteed stopping criterion, the Constant-Support Assumption for existence and tractability of optimal policies, and a convergent method to handle end components via collapsing techniques, enabling scalable performance on models with up to millions of states. Experiments demonstrate significant speedups over explicit-SG approaches and show the method’s applicability to complex uncertainty sets like $L^2$-balls, with practical runtimes on large-scale benchmarks. The framework thus delivers a generic, reliable, and efficient tool for robust optimization in sequential decision-making under uncertainty, with broad applicability in domains requiring certified guarantees.
Abstract
Markov decision processes (MDP) are a well-established model for sequential decision-making in the presence of probabilities. In robust MDP (RMDP), every action is associated with an uncertainty set of probability distributions, modelling that transition probabilities are not known precisely. Based on the known theoretical connection to stochastic games, we provide a framework for solving RMDPs that is generic, reliable, and efficient. It is *generic* both with respect to the model, allowing for a wide range of uncertainty sets, including but not limited to intervals, $L^1$- or $L^2$-balls, and polytopes; and with respect to the objective, including long-run average reward, undiscounted total reward, and stochastic shortest path. It is *reliable*, as our approach not only converges in the limit, but provides precision guarantees at any time during the computation. It is *efficient* because -- in contrast to state-of-the-art approaches -- it avoids explicitly constructing the underlying stochastic game. Consequently, our prototype implementation outperforms existing tools by several orders of magnitude and can solve RMDPs with a million states in under a minute.