ODRL: A Benchmark for Off-Dynamics Reinforcement Learning

TL;DR

ODRL is introduced, the first benchmark tailored for evaluating off-dynamics RL methods and contains four experimental settings where the source and target domains can be either online or offline, and provides diverse tasks and a broad spectrum of dynamics shifts, making it a reliable platform to comprehensively evaluate the agent's adaptation ability to the target domain.

Abstract

We consider off-dynamics reinforcement learning (RL) where one needs to transfer policies across different domains with dynamics mismatch. Despite the focus on developing dynamics-aware algorithms, this field is hindered due to the lack of a standard benchmark. To bridge this gap, we introduce ODRL, the first benchmark tailored for evaluating off-dynamics RL methods. ODRL contains four experimental settings where the source and target domains can be either online or offline, and provides diverse tasks and a broad spectrum of dynamics shifts, making it a reliable platform to comprehensively evaluate the agent's adaptation ability to the target domain. Furthermore, ODRL includes recent off-dynamics RL algorithms in a unified framework and introduces some extra baselines for different settings, all implemented in a single-file manner. To unpack the true adaptation capability of existing methods, we conduct extensive benchmarking experiments, which show that no method has universal advantages across varied dynamics shifts. We hope this benchmark can serve as a cornerstone for future research endeavors. Our code is publicly available at https://github.com/OffDynamicsRL/off-dynamics-rl.

Figures (20)

• Figure 1: An overview of selected benchmark tasks. ODRL includes multiple domains with various types of dynamics shifts, making it a reliable platform for evaluating policy adaptation ability.
• Figure 2: Benchmark setting and algorithmic implementations. Our benchmark involves 4 varied experimental settings, where the source domain and the target domain can be designated as online or offline. Numerous baselines and off-dynamics RL algorithms are implemented for each setting.
• Figure 3: Radar chart comparison of different methods. We report the aggregated normalized score of tasks within each category given the online source domain and target domain.
• Figure 4: Normalized score comparison of methods under distinct source domain datasets. We report the final average normalized score in the target domain, along with the standard deviation.
• Figure 5: Normalized score comparison of baselines given varied qualities of target domain datasets. The final mean performance and its standard deviation in the target domain are presented.

Theorems & Definitions (1)

• Definition 1: Off-dynamics RL setting