Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

More Robust Doubly Robust Off-policy Evaluation

Mehrdad Farajtabar, Yinlam Chow, Mohammad Ghavamzadeh

TL;DR

This paper introduces More Robust Doubly Robust (MRDR) off-policy evaluation, a variance-minimizing refinement of the standard Doubly Robust estimator. By learning the direct model component to minimize the DR estimator's variance, MRDR achieves strong consistency and asymptotic optimality, improving accuracy in both contextual bandits and reinforcement learning. The authors derive tractable, sample-based gradient formulas for MRDR in bandits and RL, prove theoretical guarantees, and demonstrate empirical gains over DM, IS, and DR baselines across diverse domains. This work offers a practically effective OPE approach with solid theoretical backing and broad applicability to policy evaluation under distribution shift.

Abstract

We study the problem of off-policy evaluation (OPE) in reinforcement learning (RL), where the goal is to estimate the performance of a policy from the data generated by another policy(ies). In particular, we focus on the doubly robust (DR) estimators that consist of an importance sampling (IS) component and a performance model, and utilize the low (or zero) bias of IS and low variance of the model at the same time. Although the accuracy of the model has a huge impact on the overall performance of DR, most of the work on using the DR estimators in OPE has been focused on improving the IS part, and not much on how to learn the model. In this paper, we propose alternative DR estimators, called more robust doubly robust (MRDR), that learn the model parameter by minimizing the variance of the DR estimator. We first present a formulation for learning the DR model in RL. We then derive formulas for the variance of the DR estimator in both contextual bandits and RL, such that their gradients w.r.t.~the model parameters can be estimated from the samples, and propose methods to efficiently minimize the variance. We prove that the MRDR estimators are strongly consistent and asymptotically optimal. Finally, we evaluate MRDR in bandits and RL benchmark problems, and compare its performance with the existing methods.

More Robust Doubly Robust Off-policy Evaluation

TL;DR

This paper introduces More Robust Doubly Robust (MRDR) off-policy evaluation, a variance-minimizing refinement of the standard Doubly Robust estimator. By learning the direct model component to minimize the DR estimator's variance, MRDR achieves strong consistency and asymptotic optimality, improving accuracy in both contextual bandits and reinforcement learning. The authors derive tractable, sample-based gradient formulas for MRDR in bandits and RL, prove theoretical guarantees, and demonstrate empirical gains over DM, IS, and DR baselines across diverse domains. This work offers a practically effective OPE approach with solid theoretical backing and broad applicability to policy evaluation under distribution shift.

Abstract

We study the problem of off-policy evaluation (OPE) in reinforcement learning (RL), where the goal is to estimate the performance of a policy from the data generated by another policy(ies). In particular, we focus on the doubly robust (DR) estimators that consist of an importance sampling (IS) component and a performance model, and utilize the low (or zero) bias of IS and low variance of the model at the same time. Although the accuracy of the model has a huge impact on the overall performance of DR, most of the work on using the DR estimators in OPE has been focused on improving the IS part, and not much on how to learn the model. In this paper, we propose alternative DR estimators, called more robust doubly robust (MRDR), that learn the model parameter by minimizing the variance of the DR estimator. We first present a formulation for learning the DR model in RL. We then derive formulas for the variance of the DR estimator in both contextual bandits and RL, such that their gradients w.r.t.~the model parameters can be estimated from the samples, and propose methods to efficiently minimize the variance. We prove that the MRDR estimators are strongly consistent and asymptotically optimal. Finally, we evaluate MRDR in bandits and RL benchmark problems, and compare its performance with the existing methods.

Paper Structure

This paper contains 39 sections, 9 theorems, 90 equations, 1 figure, 25 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Markov Decision Processes
  4. Off-policy Evaluation Problem
  5. Existing Approaches to OPE
  6. Direct Estimators
  7. Importance Sampling Estimators
  8. Doubly Robust Estimators
  9. More Robust Doubly Robust Estimators
  10. MRDR Estimators for Contextual Bandits
  11. MRDR Estimators for Reinforcement Learning
  12. Other Properties of the MRDR Estimators
  13. Strong Consistency
  14. Asymptotic Optimality
  15. MRDR Extensions
  16. ...and 24 more sections

Key Result

Proposition 1

The bias and variance of the DR estimator eq:DR-bandit for stochastic $\pi_e$ may be written as

Figures (1)

  • Figure 1: Environments from Thomas16DE. Top left: ModelFail; Bottom left: ModelWin; Right: Maze

Theorems & Definitions (20)

  • Proposition 1
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Proposition 2
  • proof
  • Proposition 3
  • proof
  • ...and 10 more