Robust Q-Learning for finite ambiguity sets
Cécile Decker, Julian Sester
TL;DR
This work addresses distributionally robust MDPs with finite, arbitrarily shaped ambiguity sets for transition probabilities, introducing a novel robust Q-learning algorithm that optimizes against the worst-case transition law and accommodates time-varying dynamics. The authors prove almost-sure convergence of the learned Q-values $Q_t(x,a)$ to the robust optimum $Q^*(x,a)$ under standard stochastic approximation conditions and demonstrate tractability through numerical experiments on Coin Toss and Stock investing tasks. Extensions to infinite ambiguity sets via finite approximations and to continuous state spaces via function approximation are developed, broadening applicability. The results enable practitioners to tailor ambiguity sets to specific scenarios beyond Wasserstein/KL-ball approaches, offering practical robust planning in uncertain environments.
Abstract
In this paper we propose a novel $Q$-learning algorithm allowing to solve distributionally robust Markov decision problems for which the ambiguity set of probability measures can be chosen arbitrarily as long as it comprises only a finite amount of measures. Therefore, our approach goes beyond the well-studied cases involving ambiguity sets of balls around some reference measure with the distance to reference measure being measured with respect to the Wasserstein distance or the Kullback--Leibler divergence. Hence, our approach allows the applicant to create ambiguity sets better tailored to her needs and to solve the associated robust Markov decision problem via a $Q$-learning algorithm whose convergence is guaranteed by our main result. Moreover, we showcase in several numerical experiments the tractability of our approach.