An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines

TL;DR

<3-5 sentence high-level summary> The paper investigates whether offline RL algorithms can meaningfully accelerate tabula rasa online RL by integrating offline subroutines into the online process. It formalizes a universal framework for online-with-offline subroutines and systematically evaluates multiple variants, including offline-only recommendations and offline learning followed by online fine-tuning. Key findings show that offline subroutines can substantially improve performance in environment-specific, particularly sparse-reward, tasks when paired with validation and careful data handling, while online fine-tuning methods often underperform within practical budgets. The work highlights critical failure modes and suggests directions for improving data preparation, validation, and fine-tuning strategies in online settings lacking pre-collected offline data.

Abstract

We take the novel perspective of incorporating offline RL algorithms as subroutines of tabula rasa online RL. This is feasible because an online learning agent can repurpose its historical interactions as offline dataset. We formalize this idea into a framework that accommodates several variants of offline RL incorporation such as final policy recommendation and online fine-tuning. We further introduce convenient techniques to improve its effectiveness in enhancing online learning efficiency. Our extensive and systematic empirical analyses show that 1) the effectiveness of the proposed framework depends strongly on the nature of the task, 2) our proposed techniques greatly enhance its effectiveness, and 3) existing online fine-tuning methods are overall ineffective, calling for more research therein.

Figures (8)

• Figure 1: Schematic of our incorporation of offline RL and online fine-tuning as online subroutines.
• Figure 2: Learning curves of the purely online RL process of SAC$@$1500K on all environments.
• Figure 3: Extending IQL-based fine-tuning to 1000K steps in Hopper-v2. The dashed lines indicate the original fine-tuning budgets as reported in Table \ref{['tab:fine_tuning_main']}.
• Figure 4: Distributions of episodic returns (normalized) in CalQL(SAC)'s dataset and D4RL's hopper-medium-v2.
• Figure 5: Learning curves of SAC in Mujoco (Sparse) environments with a budget of 1500K steps: (a) learning curves of SAC in Mujoco environments with different sparse rewards; (b) learning curves of SAC in selected Mujoco (Sparse) environments where scores are calculated by normalizing policies' returns using a randomly policy as a score of 0 and the policy by SAC$@\text{1500K}$ in the original Mujoco environment as the score of 100.