Benchmarks for Deep Off-Policy Evaluation

TL;DR

The paper introduces the DOPE benchmark for deep off-policy evaluation, providing high-dimensional, long-horizon tasks across two domains (RL Unplugged and D4RL) to assess policy evaluation and selection from offline data. It proposes a flexible evaluation protocol with ground-truth policy values and metrics including Absolute Error, Regret@k, and Rank correlation, and evaluates six baseline OPE methods (FQE, MB, IS, DR, DICE, VPM). Results show no method achieves oracle performance; model-based and FQE-based approaches outperform importance sampling and weight-based methods under most conditions, while data distribution and domain (especially AntMaze and D4RL) strongly affect performance. By releasing data, code, and standardized protocols, DOPE aims to standardize progress in OPE and spur development of robust, scalable offline evaluation methods.

Abstract

Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both evaluating and selecting complex policies for decision making. The ability to learn offline is particularly important in many real-world domains, such as in healthcare, recommender systems, or robotics, where online data collection is an expensive and potentially dangerous process. Being able to accurately evaluate and select high-performing policies without requiring online interaction could yield significant benefits in safety, time, and cost for these applications. While many OPE methods have been proposed in recent years, comparing results between papers is difficult because currently there is a lack of a comprehensive and unified benchmark, and measuring algorithmic progress has been challenging due to the lack of difficult evaluation tasks. In order to address this gap, we present a collection of policies that in conjunction with existing offline datasets can be used for benchmarking off-policy evaluation. Our tasks include a range of challenging high-dimensional continuous control problems, with wide selections of datasets and policies for performing policy selection. The goal of our benchmark is to provide a standardized measure of progress that is motivated from a set of principles designed to challenge and test the limits of existing OPE methods. We perform an evaluation of state-of-the-art algorithms and provide open-source access to our data and code to foster future research in this area.

Figures (25)

• Figure 1: In Off-Policy Evaluation (top) the goal is to estimate the value of a single policy given only data. Offline Policy Selection (bottom) is a closely related problem: given a set of N policies, attempt to pick the best given only data.
• Figure 2: Error is a natural measure for off-policy evaluation. However for policy selection, it is sufficient to (i) rank the policies as measured by rank correlation, or (ii) select a policy with the lowest regret.
• Figure 3: Online evaluation of policy checkpoints for 4 Offline RL algorithms with 3 random seeds. We observe a large degree of variability between the behavior of algorithms on different tasks. Without online evaluation, tuning the hyperparameters (e.g., choice of Offline RL algorithm and policy checkpoint) is challenging. This highlights the practical importance of Offline policy selection when online evaluation is not feasible. See Figure \ref{['fig:5gt-rlu']} for additional tasks.
• Figure 4: DOPE RL Unplugged Mean overall performance of baselines.
• Figure 5: DOPE D4RL Mean overall performance of baselines.