Provably Good Batch Reinforcement Learning Without Great Exploration
Yao Liu, Adith Swaminathan, Alekh Agarwal, Emma Brunskill
TL;DR
The paper tackles offline batch reinforcement learning with function approximation by enforcing conservatism in Bellman backups only over state-action pairs sufficiently covered by batch data. It introduces the Marginalized Behavior Supported (MBS) family, comprising MBS-PI and MBS-QI, which rely on a ζ-filter derived from a density estimate and a data-driven threshold b to guarantee near-optimality within the data-supported policy set Π_C^all. Theoretical results provide finite-sample bounds showing the learned policy’s value is close to the best policy that is well-represented in the batch, scaling with C = U/b and standard horizon factors, without requiring full concentrability. Empirically, MBS methods outperform strong batch baselines in CartPole (discrete) and Hopper (continuous), with practical guidance on choosing b and handling density-estimation error. This work advances safe, reliable batch RL under realistic data limitations and function approximation.
Abstract
Batch reinforcement learning (RL) is important to apply RL algorithms to many high stakes tasks. Doing batch RL in a way that yields a reliable new policy in large domains is challenging: a new decision policy may visit states and actions outside the support of the batch data, and function approximation and optimization with limited samples can further increase the potential of learning policies with overly optimistic estimates of their future performance. Recent algorithms have shown promise but can still be overly optimistic in their expected outcomes. Theoretical work that provides strong guarantees on the performance of the output policy relies on a strong concentrability assumption, that makes it unsuitable for cases where the ratio between state-action distributions of behavior policy and some candidate policies is large. This is because in the traditional analysis, the error bound scales up with this ratio. We show that a small modification to Bellman optimality and evaluation back-up to take a more conservative update can have much stronger guarantees. In certain settings, they can find the approximately best policy within the state-action space explored by the batch data, without requiring a priori assumptions of concentrability. We highlight the necessity of our conservative update and the limitations of previous algorithms and analyses by illustrative MDP examples, and demonstrate an empirical comparison of our algorithm and other state-of-the-art batch RL baselines in standard benchmarks.