Efficient Solution and Learning of Robust Factored MDPs

TL;DR

This paper addresses learning and planning under epistemic uncertainty in large-scale, factored MDPs by introducing robust factored MDPs (rf-MDPs) that factor uncertainty across state components. It shows that robust planning can be reformulated as tractable linear programs when marginal uncertainty sets are polytopes, while offering tight relaxations (interval-arithmetic, McCormick, and $L_p$-based) to scale to realistic problem sizes. The authors develop principled, finite-sample learning methods that construct marginal uncertainty sets from data and provide PAC-style guarantees that the learned robust policy performs well on the true unknown rf-MDP. Experiments demonstrate substantial sample-efficiency gains from exploiting the factored structure and the practicality of the proposed relaxations, with McCormick relaxations often providing the best trade-off between tightness and computation. These advances enable robust policy synthesis and data-efficient learning in complex, uncertain environments typical of real-world decision-making systems.

Abstract

Robust Markov decision processes (r-MDPs) extend MDPs by explicitly modelling epistemic uncertainty about transition dynamics. Learning r-MDPs from interactions with an unknown environment enables the synthesis of robust policies with provable (PAC) guarantees on performance, but this can require a large number of sample interactions. We propose novel methods for solving and learning r-MDPs based on factored state-space representations that leverage the independence between model uncertainty across system components. Although policy synthesis for factored r-MDPs leads to hard, non-convex optimisation problems, we show how to reformulate these into tractable linear programs. Building on these, we also propose methods to learn factored model representations directly. Our experimental results show that exploiting factored structure can yield dimensional gains in sample efficiency, producing more effective robust policies with tighter performance guarantees than state-of-the-art methods.

Figures (4)

• Figure 1: Part (a) shows two factors of an rf-MDP, with convex marginal uncertainty sets $\mathcal{P}$ and $\mathcal{Q}$, which are line segments in the two-dimensional probability simplex. The resulting product uncertainty set $\mathcal{P} \otimes \mathcal{Q}$ in (b) is non-convex.
• Figure 2: Projections of the interval-arithmetic (blue) and McCormick (pink) relaxations for the product of box-type uncertainty sets (coloured curve). The McCormick relaxation is tighter and has fewer spurious extreme distributions.
• Figure 3: Results for robust policy learning. The plots show objective value against processed fixed-length trajectories. Dashed curves show the robust guarantee for the learned robust policy, solid curves show its actual performance on the true model. The complete experimental results, including additional domains and total runtimes, are provided in Figure \ref{['fig:learning_extended']} of Appendix \ref{['app:experiments']}.
• Figure 4: Extended results for robust policy learning. For each benchmark, the upper plots compare performance and guarantees, and the lower plots show the total runtimes of each method.

Theorems & Definitions (10)

• Example 1
• Example 2
• Theorem 1
• Example 2
• Theorem 2
• Theorem 3
• Theorem 1: Restated
• proof
• Theorem 2: Restated
• proof