Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators

Allen Nie, Yash Chandak, Christina J. Yuan, Anirudhan Badrinath, Yannis Flet-Berliac, Emma Brunskil

TL;DR

A new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure is proposed and it is proved that the estimator is consistent and satisfies several desirable properties for policy evaluation.

Abstract

Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating a new policy online without a confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, especially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfortunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, we demonstrate that when compared to alternative approaches, our estimator can be used to select higher-performing policies in healthcare and robotics. Our work contributes to improving ease of use for a general-purpose, estimator-agnostic, off-policy evaluation framework for offline RL.

OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators

TL;DR

A new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure is proposed and it is proved that the estimator is consistent and satisfies several desirable properties for policy evaluation.

Abstract

Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from other policies. Evaluating a new policy online without a confident estimate of its performance can lead to costly, unsafe, or hazardous outcomes, especially in education and healthcare. Several OPE estimators have been proposed in the last decade, many of which have hyperparameters and require training. Unfortunately, choosing the best OPE algorithm for each task and domain is still unclear. In this paper, we propose a new algorithm that adaptively blends a set of OPE estimators given a dataset without relying on an explicit selection using a statistical procedure. We prove that our estimator is consistent and satisfies several desirable properties for policy evaluation. Additionally, we demonstrate that when compared to alternative approaches, our estimator can be used to select higher-performing policies in healthcare and robotics. Our work contributes to improving ease of use for a general-purpose, estimator-agnostic, off-policy evaluation framework for offline RL.
Paper Structure (44 sections, 4 theorems, 38 equations, 5 figures, 14 tables, 1 algorithm)

This paper contains 44 sections, 4 theorems, 38 equations, 5 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Offline policy evaluation
  4. OPE with multiple estimators
  5. Bootstrapping for model selection
  6. Notation and Problem Setup
  7. OPERA
  8. Estimating $A$:
  9. Estimating $\alpha^*$:
  10. Properties of OPERA
  11. Performance Improvement
  12. Consistency
  13. Interpretability
  14. Experiment
  15. Task/Domains
  16. ...and 29 more sections

Key Result

Theorem 1

Assume given $n$ samples in dataset $D$, and let $\Delta_c \coloneqq \mathbb E_{D_n}[*]{(*){\bar{V}^{\pi} - V^{\pi}}^2}$, there exists a $\lambda > 0$ such that

Figures (5)

  • Figure 1: Interpreting weights for different estimators. X-axis shows the value of $\hat{V}^{\pi}_1$ and Y-axis shows the value of $\hat{V}^{\pi}_2$.
  • Figure 2: Left: Results for contextual bandits. (a) MSE of estimators when the dataset size grows. (b) CDF of normalized MSE across 180 conditions by the worst MSE of that condition. Better methods lie in the top-left quadrant. Right: (c) For an MDP domain (Sepsis), we show that as dataset sizes increase, our bootstrap estimation of MSE approaches true MSE for each OPE estimator.
  • Figure 3: OPERA framework as a two-stage process.
  • Figure 4: Properties of OPERA
  • Figure : OPERA with Bootstrap

Theorems & Definitions (7)

  • Remark 1
  • Theorem 1: Finite Sample Analysis
  • Theorem 2: Performance improvement
  • Theorem 3: Invariance
  • proof
  • Theorem 4: Performance improvement
  • proof