A Finite Sample Complexity Bound for Distributionally Robust Q-learning
Shengbo Wang, Nian Si, Jose Blanchet, Zhengyuan Zhou
TL;DR
This paper addresses the challenge of distributional shifts between training simulators and deployment environments in reinforcement learning by developing a model-free, distributionally robust Q-learning algorithm with a finite-sample complexity guarantee. It extends a prior MLMC-based DR Bellman estimator to ensure a constant expected number of samples per iteration, proving unbiasedness and variance bounds within a stochastic-approximation framework. The main result is a finite-sample bound of $\tilde{O}(|S||A|(1-\gamma)^{-5}\epsilon^{-2}p_{\wedge}^{-6}\delta^{-4})$ for learning the robust $Q$-function to accuracy $\epsilon$, with the sample complexity scaling tight in $|S||A|$ and nearly tight in the effective horizon, plus empirical validation on hard MDPs and inventory control problems. The work delivers the first model-free finite-sample guarantee for distributionally robust RL and provides practical insights into step-size choices and estimator design for robust deployment. Overall, this advances robust RL by offering tractable, theory-backed guarantees and demonstrating improved robustness in simulation studies, enabling safer transfer to real-world settings.
Abstract
We consider a reinforcement learning setting in which the deployment environment is different from the training environment. Applying a robust Markov decision processes formulation, we extend the distributionally robust $Q$-learning framework studied in Liu et al. [2022]. Further, we improve the design and analysis of their multi-level Monte Carlo estimator. Assuming access to a simulator, we prove that the worst-case expected sample complexity of our algorithm to learn the optimal robust $Q$-function within an $ε$ error in the sup norm is upper bounded by $\tilde O(|S||A|(1-γ)^{-5}ε^{-2}p_{\wedge}^{-6}δ^{-4})$, where $γ$ is the discount rate, $p_{\wedge}$ is the non-zero minimal support probability of the transition kernels and $δ$ is the uncertainty size. This is the first sample complexity result for the model-free robust RL problem. Simulation studies further validate our theoretical results.