Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning

TL;DR

The paper tackles offline RL data limitations by advocating a data-centric approach, ExORL, that collects reward-free exploratory data, relabels it with downstream rewards, and learns entirely offline. It demonstrates that diverse exploratory data can enable vanilla off-policy algorithms to outperform specialized offline RL methods and supports multi-task offline RL from a single dataset. The work provides extensive empirical evidence across multiple environments and discusses how data collection choices shape performance, offering a public dataset suite to foster further research. This shifts the focus from algorithmic novelty alone to careful data generation as a key lever for offline RL success.

Abstract

Recent progress in deep learning has relied on access to large and diverse datasets. Such data-driven progress has been less evident in offline reinforcement learning (RL), because offline RL data is usually collected to optimize specific target tasks limiting the data's diversity. In this work, we propose Exploratory data for Offline RL (ExORL), a data-centric approach to offline RL. ExORL first generates data with unsupervised reward-free exploration, then relabels this data with a downstream reward before training a policy with offline RL. We find that exploratory data allows vanilla off-policy RL algorithms, without any offline-specific modifications, to outperform or match state-of-the-art offline RL algorithms on downstream tasks. Our findings suggest that data generation is as important as algorithmic advances for offline RL and hence requires careful consideration from the community. Code and data can be found at https://github.com/denisyarats/exorl .

Figures (11)

• Figure 1: Summary of the ExORL framework.
• Figure 2: Our ExORL framework exploratory data collection for offline RL consists of three phases: collect(a), relabel and learn(b). Here we showcase our method on a pointmass maze environment to demonstrate that it is possible to collect exploratory data in a completely unsupervised manner that enables effective multi-task offline RL.
• Figure 3: Offline evaluation of unsupervised datasets on one task for each of three different domains. Here we choose four representative unsupervised exploration algorithms, for full results see \ref{['section:mult_env_full']}. Vanilla TD3 usually outperforms all three offline RL algorithms.
• Figure 4: Offline evaluation of datasets for the Walker environment under three different rewards (Stand, Walk, and Run). We observe that ExORL allows for data relabeling to enable multi-task offline RL. See \ref{['section:mult_task_full']} for full results across all exploration algorithms.
• Figure 5: A comparison of three different data-collection strategies: unsupervised (intrinsic reward only), semi-supervised (intrinsic reward plus data task reward), and supervised (data task reward only). Data diversity is key to enable more aggressive trajectory stitching and task transfer.