Sample Complexity of Variance-reduced Distributionally Robust Q-learning
Shengbo Wang, Nian Si, Jose Blanchet, Zhengyuan Zhou
TL;DR
The paper addresses robust reinforcement learning under distributional shifts by formulating KL (and later χ²) divergence ambiguity sets around nominal environment dynamics and rewards. It proposes two model-free algorithms, DR Q-learning and its Variance-Reduced variant, and proves near-optimal, δ-independent minimax sample complexity bounds, with VR DR Q-learning achieving the best known horizon- and ε−dependent rates. The work shows that as δ→0 the complexity does not blow up, connecting robust and non-robust RL and enabling efficient learning under small adversarial power. Empirical results on hard MDPs corroborate the theory, demonstrating that variance reduction yields faster convergence and robustness to distributional shifts in practice, and the χ² extension broadens applicability to alternative ambiguity sets.
Abstract
Dynamic decision-making under distributional shifts is of fundamental interest in theory and applications of reinforcement learning: The distribution of the environment in which the data is collected can differ from that of the environment in which the model is deployed. This paper presents two novel model-free algorithms, namely the distributionally robust Q-learning and its variance-reduced counterpart, that can effectively learn a robust policy despite distributional shifts. These algorithms are designed to efficiently approximate the $q$-function of an infinite-horizon $γ$-discounted robust Markov decision process with Kullback-Leibler ambiguity set to an entry-wise $ε$-degree of precision. Further, the variance-reduced distributionally robust Q-learning combines the synchronous Q-learning with variance-reduction techniques to enhance its performance. Consequently, we establish that it attains a minimax sample complexity upper bound of $\tilde O(|\mathbf{S}||\mathbf{A}|(1-γ)^{-4}ε^{-2})$, where $\mathbf{S}$ and $\mathbf{A}$ denote the state and action spaces. This is the first complexity result that is independent of the ambiguity size $δ$, thereby providing new complexity theoretic insights. Additionally, a series of numerical experiments confirm the theoretical findings and the efficiency of the algorithms in handling distributional shifts.