Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approach
Dengwang Tang, Rahul Jain, Botao Hao, Zheng Wen
TL;DR
This work tackles online reinforcement learning in infinite-horizon MDPs with an available offline dataset generated by an imperfect expert. It introduces an ideal Bayesian PSRL framework (inf-iPSRL) and a practical bootstrap-style variant (inf-iRLSVI) that leverage offline demonstrations through a competence-aware model of the expert. The authors derive a prior-dependent regret bound showing how offline data quality influences performance and show that strong demonstrations can yield near-constant regret as data grows. They further connect online RL with imitation learning, offering a principled way to fuse offline and online information and suggesting robust directions for universal RL that unify online, offline, and imitation paradigms. The results provide a quantitative understanding of data efficiency gains from offline sources and practical algorithms to realize them in infinite-horizon settings.
Abstract
In this paper, we study the problem of efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with. We assume that the offline dataset is generated by an expert but with unknown level of competence, i.e., it is not perfect and not necessarily using the optimal policy. We show that if the learning agent models the behavioral policy (parameterized by a competence parameter) used by the expert, it can do substantially better in terms of minimizing cumulative regret, than if it doesn't do that. We establish an upper bound on regret of the exact informed PSRL algorithm that scales as $\tilde{O}(\sqrt{T})$. This requires a novel prior-dependent regret analysis of Bayesian online learning algorithms for the infinite horizon setting. We then propose the Informed RLSVI algorithm to efficiently approximate the iPSRL algorithm.