Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs via Approximation by Discounted-Reward MDPs
Kihyuk Hong, Woojin Chae, Yufan Zhang, Dabeen Lee, Ambuj Tewari
TL;DR
This work tackles infinite-horizon average-reward reinforcement learning in linear MDPs, where the non-contractive Bellman operator complicates learning. The authors propose a novel reduction to discounted RL with a carefully chosen discount factor $\gamma$ and an optimistic value-iteration scheme, augmented by a nonstationary planning strategy and a value-function clipping technique to achieve near-optimal regret. The resulting algorithm, called $\gamma$-LSCVI-UCB, runs in polynomial time in problem parameters and attains a regret bound of $\tilde{O}(\text{sp}(v^*) \sqrt{d^3 T \log(dT/\delta)})$ without requiring ergodicity, significantly advancing practical applicability. The approach also introduces planning-before-action via pregenerated $Q$-functions and controlled restarts when information doubles, enabling efficient data utilization and sample efficiency with linear function approximation.
Abstract
We study the problem of infinite-horizon average-reward reinforcement learning with linear Markov decision processes (MDPs). The associated Bellman operator of the problem not being a contraction makes the algorithm design challenging. Previous approaches either suffer from computational inefficiency or require strong assumptions on dynamics, such as ergodicity, for achieving a regret bound of $\widetilde{O}(\sqrt{T})$. In this paper, we propose the first algorithm that achieves $\widetilde{O}(\sqrt{T})$ regret with computational complexity polynomial in the problem parameters, without making strong assumptions on dynamics. Our approach approximates the average-reward setting by a discounted MDP with a carefully chosen discounting factor, and then applies an optimistic value iteration. We propose an algorithmic structure that plans for a nonstationary policy through optimistic value iteration and follows that policy until a specified information metric in the collected data doubles. Additionally, we introduce a value function clipping procedure for limiting the span of the value function for sample efficiency.