Towards Minimax Optimality of Model-based Robust Reinforcement Learning
Pierre Clavier, Erwan Le Pennec, Matthieu Geist
TL;DR
The paper analyzes sample complexity for model-based robust reinforcement learning in Robust MDPs with $L_p$-ball uncertainty sets under a generative-model framework. It derives upper bounds showing $ ilde{\mathcal{O}}(H^4|S||A|/\epsilon^2)$ in the general case and improves to $ ilde{\mathcal{O}}(H^3|S||A|/\epsilon^2)$ when the uncertainty radius is small, achieving minimax-optimality relative to non-robust bounds. The authors introduce Distributionally Robust Value Iteration for $L_p$ norms ($\\mathtt{DRVI \\,L_P}$) and develop a duality-based analysis combined with Bernstein concentration and absorbing MDP techniques to tighten bounds. These results advance the understanding of when robustness does not incur extra sample complexity and identify regimes where robust planning attains minimax-optimal performance with generative-model access.
Abstract
We study the sample complexity of obtaining an $ε$-optimal policy in \emph{Robust} discounted Markov Decision Processes (RMDPs), given only access to a generative model of the nominal kernel. This problem is widely studied in the non-robust case, and it is known that any planning approach applied to an empirical MDP estimated with $\tilde{\mathcal{O}}(\frac{H^3 \mid S \mid\mid A \mid}{ε^2})$ samples provides an $ε$-optimal policy, which is minimax optimal. Results in the robust case are much more scarce. For $sa$- (resp $s$-)rectangular uncertainty sets, the best known sample complexity is $\tilde{\mathcal{O}}(\frac{H^4 \mid S \mid^2\mid A \mid}{ε^2})$ (resp. $\tilde{\mathcal{O}}(\frac{H^4 \mid S \mid^2\mid A \mid^2}{ε^2})$), for specific algorithms and when the uncertainty set is based on the total variation (TV), the KL or the Chi-square divergences. In this paper, we consider uncertainty sets defined with an $L_p$-ball (recovering the TV case), and study the sample complexity of \emph{any} planning algorithm (with high accuracy guarantee on the solution) applied to an empirical RMDP estimated using the generative model. In the general case, we prove a sample complexity of $\tilde{\mathcal{O}}(\frac{H^4 \mid S \mid\mid A \mid}{ε^2})$ for both the $sa$- and $s$-rectangular cases (improvements of $\mid S \mid$ and $\mid S \mid\mid A \mid$ respectively). When the size of the uncertainty is small enough, we improve the sample complexity to $\tilde{\mathcal{O}}(\frac{H^3 \mid S \mid\mid A \mid }{ε^2})$, recovering the lower-bound for the non-robust case for the first time and a robust lower-bound when the size of the uncertainty is small enough.