Data- and Variance-dependent Regret Bounds for Online Tabular MDPs
Mingyi Li, Taira Tsuchiya, Kenji Yamanishi
TL;DR
The paper addresses online learning in finite-horizon, episodic, tabular MDPs with known transitions, introducing best-of-both-worlds algorithms that adapt to data-dependent complexities in adversarial settings and variance-dependent statistics in stochastic settings. It develops two OFTRL-based frameworks: global optimization over occupancy measures and per-state policy optimization, each augmented with log-barrier regularization, loss-prediction schemes, and novel estimators (including a dilated bonus and an optimistic Q-function) to achieve refined regret bounds. The work introduces first-order, second-order, path-length, and variance-based complexity measures ($L^\star$, $Q_\infty$, $V_1$, $\mathbb{V}$, $\mathbb{V}^c(s)$) to capture problem structure, and proves matching lower bounds $\Omega(\sqrt{SAL^\star})$, $\Omega(\sqrt{SAQ_\infty})$, $\Omega(\sqrt{HV_1})$, and $\Omega(\sqrt{SA\mathbb{V} T})$, showing near-optimality of the proposed guarantees. The results yield data- and variance-aware regret that scales with these complexities, including polylog factors and full adaptivity across regimes, paving the way for robust online MDP methods with theoretical optimality in both adversarial and stochastic environments.
Abstract
This work studies online episodic tabular Markov decision processes (MDPs) with known transitions and develops best-of-both-worlds algorithms that achieve refined data-dependent regret bounds in the adversarial regime and variance-dependent regret bounds in the stochastic regime. We quantify MDP complexity using a first-order quantity and several new data-dependent measures for the adversarial regime, including a second-order quantity and a path-length measure, as well as variance-based measures for the stochastic regime. To adapt to these measures, we develop algorithms based on global optimization and policy optimization, both built on optimistic follow-the-regularized-leader with log-barrier regularization. For global optimization, our algorithms achieve first-order, second-order, and path-length regret bounds in the adversarial regime, and in the stochastic regime, they achieve a variance-aware gap-independent bound and a variance-aware gap-dependent bound that is polylogarithmic in the number of episodes. For policy optimization, our algorithms achieve the same data- and variance-dependent adaptivity, up to a factor of the episode horizon, by exploiting a new optimistic $Q$-function estimator. Finally, we establish regret lower bounds in terms of data-dependent complexity measures for the adversarial regime and a variance measure for the stochastic regime, implying that the regret upper bounds achieved by the global-optimization approach are nearly optimal.