Provably Efficient Exploration in Quantum Reinforcement Learning with Logarithmic Worst-Case Regret
Han Zhong, Jiachen Hu, Yecheng Xue, Tongyang Li, Liwei Wang
TL;DR
This work tackles the challenge of online exploration in reinforcement learning by enabling quantum speedups through quantum oracles. It introduces two algorithms: Quantum UCRL for tabular MDPs and Quantum UCRL-VTR for linear mixture MDPs, each with regret bounds that depend polylogarithmically on the episode count, i.e., $\mathcal{O}(\mathrm{poly}(S,A,H,\log T))$ and $\mathcal{O}(\mathrm{poly}(d,H,\log T))$ respectively. Central to the approach are quantum mean estimation and amplitude estimation subroutines, a doubling-based lazy updating scheme to reuse quantum samples, and novel feature/ regression-target estimation techniques, all enabling logarithmic-like dependence on $T$ relative to classical $\Omega(\sqrt{T})$ lower bounds. The results establish the first provable logarithmic worst-case regret for online quantum RL and point to a broader framework for extending quantum speedups to general function approximation in RL.
Abstract
While quantum reinforcement learning (RL) has attracted a surge of attention recently, its theoretical understanding is limited. In particular, it remains elusive how to design provably efficient quantum RL algorithms that can address the exploration-exploitation trade-off. To this end, we propose a novel UCRL-style algorithm that takes advantage of quantum computing for tabular Markov decision processes (MDPs) with $S$ states, $A$ actions, and horizon $H$, and establish an $\mathcal{O}(\mathrm{poly}(S, A, H, \log T))$ worst-case regret for it, where $T$ is the number of episodes. Furthermore, we extend our results to quantum RL with linear function approximation, which is capable of handling problems with large state spaces. Specifically, we develop a quantum algorithm based on value target regression (VTR) for linear mixture MDPs with $d$-dimensional linear representation and prove that it enjoys $\mathcal{O}(\mathrm{poly}(d, H, \log T))$ regret. Our algorithms are variants of UCRL/UCRL-VTR algorithms in classical RL, which also leverage a novel combination of lazy updating mechanisms and quantum estimation subroutines. This is the key to breaking the $Ω(\sqrt{T})$-regret barrier in classical RL. To the best of our knowledge, this is the first work studying the online exploration in quantum RL with provable logarithmic worst-case regret.