Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterization

Anirudh Satheesh, Ziyi Chen, Furong Huang, Heng Huang

TL;DR

The paper tackles robust MDPs under model uncertainty with general policy parameterization, addressing both $s$-rectangular and non-rectangular uncertainty and extending beyond tabular settings. It introduces an entropy-regularized discounted reduction to restore strong duality in average-reward contexts, enabling tractable minimax optimization and gradient-based methods. A novel multilevel Monte Carlo gradient estimator achieves near-optimal sample complexity for infinite-horizon gradient estimation, and the authors develop a projected gradient descent method for $s$-rectangular uncertainty and a Frank–Wolfe method for non-rectangular uncertainty, with improved rates in discounted and average-reward regimes. Importantly, the work provides the first end-to-end sample complexity guarantees for average-reward RMDPs with general parameterization, and extends robust optimization guarantees to infinite state spaces via linear parameterization. Collectively, these results advance scalable robust planning and learning for high-dimensional, uncertain environments, offering a rigorous foundation and practical algorithms for robust policies in both discounted and average-reward settings.

Abstract

We study robust Markov decision processes (RMDPs) with general policy parameterization under s-rectangular and non-rectangular uncertainty sets. Prior work is largely limited to tabular policies, and hence either lacks sample complexity guarantees or incurs high computational cost. Our method reduces the average reward RMDPs to entropy-regularized discounted robust MDPs, restoring strong duality and enabling tractable equilibrium computation. We prove novel Lipschitz and Lipschitz-smoothness properties for general policy parameterizations that extends to infinite state spaces. To address infinite-horizon gradient estimation, we introduce a multilevel Monte Carlo gradient estimator with $\tilde{\mathcal{O}}(ε^{-2})$ sample complexity, a factor of $\mathcal{O}(ε^{-2})$ improvement over prior work. Building on this, we design a projected gradient descent algorithm for s-rectangular uncertainty ($\mathcal{O}(ε^{-5})$) and a Frank--Wolfe algorithm for non-rectangular uncertainty ($\mathcal{O}(ε^{-4})$ discounted, $\mathcal{O}(ε^{-10.5})$ average reward), significantly improving prior results in both the discounted setting and average reward setting. Our work is the first one to provide sample complexity guarantees for RMDPs with general policy parameterization beyond $(s, a)$-rectangularity. It also provides the first such guarantees in the average reward setting and improves existing bounds for discounted robust MDPs.

Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterization

TL;DR

The paper tackles robust MDPs under model uncertainty with general policy parameterization, addressing both -rectangular and non-rectangular uncertainty and extending beyond tabular settings. It introduces an entropy-regularized discounted reduction to restore strong duality in average-reward contexts, enabling tractable minimax optimization and gradient-based methods. A novel multilevel Monte Carlo gradient estimator achieves near-optimal sample complexity for infinite-horizon gradient estimation, and the authors develop a projected gradient descent method for -rectangular uncertainty and a Frank–Wolfe method for non-rectangular uncertainty, with improved rates in discounted and average-reward regimes. Importantly, the work provides the first end-to-end sample complexity guarantees for average-reward RMDPs with general parameterization, and extends robust optimization guarantees to infinite state spaces via linear parameterization. Collectively, these results advance scalable robust planning and learning for high-dimensional, uncertain environments, offering a rigorous foundation and practical algorithms for robust policies in both discounted and average-reward settings.

Abstract

We study robust Markov decision processes (RMDPs) with general policy parameterization under s-rectangular and non-rectangular uncertainty sets. Prior work is largely limited to tabular policies, and hence either lacks sample complexity guarantees or incurs high computational cost. Our method reduces the average reward RMDPs to entropy-regularized discounted robust MDPs, restoring strong duality and enabling tractable equilibrium computation. We prove novel Lipschitz and Lipschitz-smoothness properties for general policy parameterizations that extends to infinite state spaces. To address infinite-horizon gradient estimation, we introduce a multilevel Monte Carlo gradient estimator with sample complexity, a factor of improvement over prior work. Building on this, we design a projected gradient descent algorithm for s-rectangular uncertainty () and a Frank--Wolfe algorithm for non-rectangular uncertainty ( discounted, average reward), significantly improving prior results in both the discounted setting and average reward setting. Our work is the first one to provide sample complexity guarantees for RMDPs with general policy parameterization beyond -rectangularity. It also provides the first such guarantees in the average reward setting and improves existing bounds for discounted robust MDPs.
Paper Structure (40 sections, 29 theorems, 149 equations, 1 table, 3 algorithms)

This paper contains 40 sections, 29 theorems, 149 equations, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Challenges and Contributions
  3. Related Work
  4. Parameterization of Robust MDPs.
  5. Average-Reward Duality Gap.
  6. Formulation
  7. Robust Average Reward MDPs
  8. On Duality and Discounted Reductions
  9. Entropy-Regularized Robust MDPs
  10. Extensions to Large and/or Infinite State Spaces
  11. Optimization of Transition Kernel Parameters
  12. Duality and $s$-Rectangularity
  13. Gradient Dominance under Linear Parameterization
  14. Regularity Properties of the Entropy-Regularized Objective
  15. Regularity of the Policy Class.
  16. ...and 25 more sections

Key Result

Lemma 3.6

Under Assumption assumption: transition kernel model error and if the value function $V_{P_\xi, \tau}^{\pi_\theta}$ is $L_V$-Lipschitz uniformly with respect to the state distance metric, for any $\epsilon \geq 0$ and $\tau > 0$ an $(\epsilon, \tau)$-Nash Equilibrium exists. If $(\theta, \xi)$ is su

Theorems & Definitions (49)

  • Definition 3.1: Typical Uncertainty Sets
  • Definition 3.5
  • Lemma 3.6
  • Lemma 4.1: Gradient Dominance under Linear Parameterization
  • Lemma 4.4
  • Lemma 4.5: Lipschitz Smoothness of the Outer Objective
  • Theorem 5.1: Convergence of Algorithm \ref{['alg: pgd']}
  • Theorem 5.2: Convergence of Algorithm \ref{['alg: frank_wolfe']}
  • Lemma 5.3: Gradient Dominance for Non-rectangular Uncertainty sets
  • Lemma 6.1: Sample Complexity of MLMC Gradient Estimator
  • ...and 39 more