Q-Learning with Fine-Grained Gap-Dependent Regret
Haochen Zhang, Zhong Zheng, Lingzhou Xue
TL;DR
This work addresses the challenge of obtaining fine-grained, gap-dependent regret bounds for model-free RL in episodic tabular MDPs by introducing a framework that separately analyzes optimal and suboptimal state-action visitation. It proves a first refined, gap-sensitive bound for a UCB-based algorithm (UCB-Hoeffding) and extends the framework to a simpler variant (ULCB-Hoeffding), both achieving improved dependence on the per-gap terms $\Delta_h(s,a)$ rather than the coarse $\Delta_{\min}$. For the non-UCB-based AMB, the authors diagnose core design and analysis flaws, proposing Refined AMB that ensures unbiased multi-step bootstrapping and valid martingale concentration, yielding the first rigorous fine-grained bound in this regime. Across synthetic experiments, Refined AMB and ULCB-Hoeffding outperform the original AMB, with UCB-Hoeffding matching or exceeding performance, and all methods displaying logarithmic regret growth in the number of episodes. These results advance understanding of problem-dependent performance in model-free RL and guide the design of gap-aware exploration strategies.
Abstract
We study fine-grained gap-dependent regret bounds for model-free reinforcement learning in episodic tabular Markov Decision Processes. Existing model-free algorithms achieve minimax worst-case regret, but their gap-dependent bounds remain coarse and fail to fully capture the structure of suboptimality gaps. We address this limitation by establishing fine-grained gap-dependent regret bounds for both UCB-based and non-UCB-based algorithms. In the UCB-based setting, we develop a novel analytical framework that explicitly separates the analysis of optimal and suboptimal state-action pairs, yielding the first fine-grained regret upper bound for UCB-Hoeffding (Jin et al., 2018). To highlight the generality of this framework, we introduce ULCB-Hoeffding, a new UCB-based algorithm inspired by AMB (Xu et al.,2021) but with a simplified structure, which enjoys fine-grained regret guarantees and empirically outperforms AMB. In the non-UCB-based setting, we revisit the only known algorithm AMB, and identify two key issues in its algorithm design and analysis: improper truncation in the $Q$-updates and violation of the martingale difference condition in its concentration argument. We propose a refined version of AMB that addresses these issues, establishing the first rigorous fine-grained gap-dependent regret for a non-UCB-based method, with experiments demonstrating improved performance over AMB.