A Clean Slate for Offline Reinforcement Learning

TL;DR

This work proposes Unifloral, a unified algorithm that encapsulates diverse prior approaches within a single, comprehensive hyperparameter space, enabling algorithm development in a shared hyperparameter space and develops two novel algorithms - TD3-AWR and MoBRAC - which substantially outperform established baselines.

Abstract

Progress in offline reinforcement learning (RL) has been impeded by ambiguous problem definitions and entangled algorithmic designs, resulting in inconsistent implementations, insufficient ablations, and unfair evaluations. Although offline RL explicitly avoids environment interaction, prior methods frequently employ extensive, undocumented online evaluation for hyperparameter tuning, complicating method comparisons. Moreover, existing reference implementations differ significantly in boilerplate code, obscuring their core algorithmic contributions. We address these challenges by first introducing a rigorous taxonomy and a transparent evaluation protocol that explicitly quantifies online tuning budgets. To resolve opaque algorithmic design, we provide clean, minimalistic, single-file implementations of various model-free and model-based offline RL methods, significantly enhancing clarity and achieving substantial speed-ups. Leveraging these streamlined implementations, we propose Unifloral, a unified algorithm that encapsulates diverse prior approaches within a single, comprehensive hyperparameter space, enabling algorithm development in a shared hyperparameter space. Using Unifloral with our rigorous evaluation protocol, we develop two novel algorithms - TD3-AWR (model-free) and MoBRAC (model-based) - which substantially outperform established baselines. Our implementation is publicly available at https://github.com/EmptyJackson/unifloral.

Figures (14)

• Figure 1: Formalizing the variants of offline RL---we define a range of offline RL variants (\ref{['sec:variants']}), with policy performance being measured post-deployment. Pre-deployment policy selection (2a) and post-deployment policy selection (2b) use a policy-selection bandit after offline training, whilst (3) uses unrestricted policy updates.
• Figure 2: Overview of our evaluation procedure. Left: We sample hyperparameters, train the corresponding policies, and collect their final evaluation scores. Right: We simulate hyperparameter tuning using the collected scores by subsampling $K$ policy scores and recording the best-arm performance of a UCB tuning bandit operating over them.
• Figure 3: Evaluation of prior algorithms---mean and 95% CI over 500 bandit rollouts, with $K=8$ policy arms subsampled from 20 trained policies each rollout. The $x$-axis denotes the number of bandit pulls, whilst the $y$-axis denotes the true expected score of the estimated best arm after $x$ pulls.
• Figure 4: Distractor policy phenomenon---we demonstrate the occurrence of distractor policies trained by ReBRAC on hopper-medium and their impact on policy evaluation.
• Figure 5: We provide clean and consistent single-file implementations, as demonstrated by compact implementations and minimal differences between algorithms.