The Challenges of Exploration for Offline Reinforcement Learning

TL;DR

The paper addresses data collection challenges in offline reinforcement learning by proposing Explore2Offline, a framework that decouples data gathering from policy learning and uses reward relabelling to evaluate data usefulness across downstream tasks. It introduces Intrinsic Model Predictive Control (IMPC) as a planning-based, curiosity-driven exploration method and assesses data quality by training offline policies with CRR on relabeled datasets. Across DM Control Suite tasks and Explore Suite challenges, larger task-agnostic datasets consistently improve offline performance, with IMPC and RND-based variants achieving strong results on large collections. The findings underscore the importance of data-centric design in offline RL and suggest that task-agnostic exploration can support multitask transfer, albeit with variability across environments and downstream tasks. The work offers a practical pathway to build and evaluate versatile datasets that support multiple downstream objectives in offline RL settings.

Abstract

Offline Reinforcement Learning (ORL) enablesus to separately study the two interlinked processes of reinforcement learning: collecting informative experience and inferring optimal behaviour. The second step has been widely studied in the offline setting, but just as critical to data-efficient RL is the collection of informative data. The task-agnostic setting for data collection, where the task is not known a priori, is of particular interest due to the possibility of collecting a single dataset and using it to solve several downstream tasks as they arise. We investigate this setting via curiosity-based intrinsic motivation, a family of exploration methods which encourage the agent to explore those states or transitions it has not yet learned to model. With Explore2Offline, we propose to evaluate the quality of collected data by transferring the collected data and inferring policies with reward relabelling and standard offline RL algorithms. We evaluate a wide variety of data collection strategies, including a new exploration agent, Intrinsic Model Predictive Control (IMPC), using this scheme and demonstrate their performance on various tasks. We use this decoupled framework to strengthen intuitions about exploration and the data prerequisites for effective offline RL.

Figures (13)

• Figure 1: The Explore2Offline framework for evaluating data-efficient intrinsic agents. First the agent acts in the environment task-agnostically to search for novel states. After a set lifetime, the agent experience stored in a replay buffer is labeled with the rewards of a task of interest. This replay buffer is used to train an RL policy with the offline reinforcement learning algorithm Critic Regularized Regression in order to finally evaluate the quality of exploration in the environment.
• Figure 2: A conceptual depiction of an exploration with foresight via planning. By simulating the future with a learned dynamics model, ideally an agent should be able to avoid damaging states before having experienced them to collect interesting data in a sample-efficient manner.
• Figure 3: Collected reward per-episode (1000 environment steps) distributions across a subset of tasks and their Explore Suite variants. Boxplots depict the median episode reward (red line), 25th and 75th percentiles (box), and the maximum and minimum reward computed over 5000 training episodes (whiskers). Only a horizontal line indicates that the observed episode reward is equal or near 0.
• Figure 4: Comparing the collected rewards of an exploration agent (left) to the reward achieved when training an offline RL algorithm on the same data (right). There is a substantial gain in performance with offline RL algorithms simply by having more data, irregardless as to if the data has higher density of observed rewards. The median, $90^\text{th}$ and $10^\text{th}$ percentiles are shown for each agent, combined across tasks. Note: the MPO agent learns online and is the only task-aware method.
• Figure 5: Correlation of final offline RL performance to cumulative reward in the training set on two example tasks and their explore-suite variants. The ORL performances are the median across 3 trials for each task and agent. As the dataset sizes we use span many orders of magnitude of samples, both axes are plotted on a log-scale. Across all tasks, there is a trend of more reward in the training distribution relating to a better performing CRR agent.