Sample-efficient Learning of Infinite-horizon Average-reward MDPs with General Function Approximation
Jianliang He, Han Zhong, Zhuoran Yang
TL;DR
This work advances sample-efficient reinforcement learning for infinite-horizon average-reward MDPs under general function approximation by introducing a unifying complexity metric, AGEC, and a flexible algorithmic framework, LOOP. LOOP combines optimistic planning with a confidence-set construction and a lazy policy-update rule, enabling it to handle both model-based and value-based representations. The authors show a sublinear regret bound $\tilde{\mathcal{O}}(\mathrm{sp}(V^*) \cdot d \sqrt{T\beta})$, where $d$ captures AGEC and log-covering complexity, thereby encompassing many tractable AMDP models including linear, kernel, and linear-Q*/V* variants. This framework provides a cohesive theoretical basis for understanding exploration in AMDPs with general function approximation and points to broad applicability in complex, high-dimensional decision problems.
Abstract
We study infinite-horizon average-reward Markov decision processes (AMDPs) in the context of general function approximation. Specifically, we propose a novel algorithmic framework named Local-fitted Optimization with OPtimism (LOOP), which incorporates both model-based and value-based incarnations. In particular, LOOP features a novel construction of confidence sets and a low-switching policy updating scheme, which are tailored to the average-reward and function approximation setting. Moreover, for AMDPs, we propose a novel complexity measure -- average-reward generalized eluder coefficient (AGEC) -- which captures the challenge of exploration in AMDPs with general function approximation. Such a complexity measure encompasses almost all previously known tractable AMDP models, such as linear AMDPs and linear mixture AMDPs, and also includes newly identified cases such as kernel AMDPs and AMDPs with Bellman eluder dimensions. Using AGEC, we prove that LOOP achieves a sublinear $\tilde{\mathcal{O}}(\mathrm{poly}(d, \mathrm{sp}(V^*)) \sqrt{Tβ} )$ regret, where $d$ and $β$ correspond to AGEC and log-covering number of the hypothesis class respectively, $\mathrm{sp}(V^*)$ is the span of the optimal state bias function, $T$ denotes the number of steps, and $\tilde{\mathcal{O}} (\cdot) $ omits logarithmic factors. When specialized to concrete AMDP models, our regret bounds are comparable to those established by the existing algorithms designed specifically for these special cases. To the best of our knowledge, this paper presents the first comprehensive theoretical framework capable of handling nearly all AMDPs.