Near-Optimal Sample Complexities of Divergence-based S-rectangular Distributionally Robust Reinforcement Learning
Zhenghao Li, Shengbo Wang, Nian Si
TL;DR
This work addresses distributional mismatches in reinforcement learning by studying divergence-based S-rectangular robust MDPs under a generative model. It develops an empirical Bellman estimator and proves near-optimal sample complexity bounds that scale linearly with the state and action spaces and polynomially with the discount factor, achieving optimal dependence on $|\mathcal{S}|$, $|\mathcal{A}|$, and $\varepsilon$ simultaneously. The analysis covers KL- and $f_k$-divergence uncertainty sets, deriving dual formulations and explicit contraction-based error bounds, with sample complexities of $\widetilde{O}(|\mathcal{S}||\mathcal{A}|(1-\gamma)^{-4}\mathfrak{p}_{\wedge}^{-1}\varepsilon^{-2})$ (KL) and similar for $f_k$. Numerical experiments on robust inventory control and lower-bound MDP instances corroborate the theory, showing fast learning rates and the predicted scaling with model size.
Abstract
Distributionally robust reinforcement learning (DR-RL) has recently gained significant attention as a principled approach that addresses discrepancies between training and testing environments. To balance robustness, conservatism, and computational traceability, the literature has introduced DR-RL models with SA-rectangular and S-rectangular adversaries. While most existing statistical analyses focus on SA-rectangular models, owing to their algorithmic simplicity and the optimality of deterministic policies, S-rectangular models more accurately capture distributional discrepancies in many real-world applications and often yield more effective robust randomized policies. In this paper, we study the empirical value iteration algorithm for divergence-based S-rectangular DR-RL and establish near-optimal sample complexity bounds of $\widetilde{O}(|\mathcal{S}||\mathcal{A}|(1-γ)^{-4}\varepsilon^{-2})$, where $\varepsilon$ is the target accuracy, $|\mathcal{S}|$ and $|\mathcal{A}|$ denote the cardinalities of the state and action spaces, and $γ$ is the discount factor. To the best of our knowledge, these are the first sample complexity results for divergence-based S-rectangular models that achieve optimal dependence on $|\mathcal{S}|$, $|\mathcal{A}|$, and $\varepsilon$ simultaneously. We further validate this theoretical dependence through numerical experiments on a robust inventory control problem and a theoretical worst-case example, demonstrating the fast learning performance of our proposed algorithm.