d3rlpy: An Offline Deep Reinforcement Learning Library

TL;DR

The paper addresses reproducibility and usability barriers in offline deep reinforcement learning by introducing d3rlpy, an open-source PyTorch-based library with a plug-and-play, fully documented API for offline and online algorithms. It details a modular design featuring MDPDataset, EncoderFactory, and QFunctionFactory to support diverse datasets (D4RL, Atari) and custom architectures, enabling seamless offline-to-online fine-tuning. A large-scale benchmark against established baselines demonstrates faithful implementation quality and provides experimental scripts and results to facilitate reproducibility. The work highlights practical impact by lowering the barrier to conducting offline RL experiments and enabling robust, comparable evaluations across studies.

Abstract

In this paper, we introduce d3rlpy, an open-sourced offline deep reinforcement learning (RL) library for Python. d3rlpy supports a set of offline deep RL algorithms as well as off-policy online algorithms via a fully documented plug-and-play API. To address a reproducibility issue, we conduct a large-scale benchmark with D4RL and Atari 2600 dataset to ensure implementation quality and provide experimental scripts and full tables of results. The d3rlpy source code can be found on GitHub: \url{https://github.com/takuseno/d3rlpy}.

Figures (1)

• Figure 1: The illustration of module components in d3rlpy. MDPDataset and OpenAI Gym-styled environment can be used to train policies.