Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Doubly Robust Policy Evaluation and Learning

Miroslav Dudik, John Langford, Lihong Li

TL;DR

The paper tackles offline policy evaluation and learning in contextual bandits with partial rewards. It introduces a doubly robust estimator that combines a reward model with a past-policy model, ensuring unbiasedness if either component is accurate and reducing variance relative to IPS. Through non-asymptotic bias and variance analyses, it shows DR often outperforms DM and IPS, particularly in off-policy evaluation and policy optimization tasks. Empirical results on benchmark multiclass classification datasets and a large-scale real-world dataset demonstrate consistent improvements in estimation accuracy and classifier performance. The work argues that the DR approach should become standard practice in contextual bandit settings due to its robustness and practical benefits.

Abstract

We study decision making in environments where the reward is only partially observed, but can be modeled as a function of an action and an observed context. This setting, known as contextual bandits, encompasses a wide variety of applications including health-care policy and Internet advertising. A central task is evaluation of a new policy given historic data consisting of contexts, actions and received rewards. The key challenge is that the past data typically does not faithfully represent proportions of actions taken by a new policy. Previous approaches rely either on models of rewards or models of the past policy. The former are plagued by a large bias whereas the latter have a large variance. In this work, we leverage the strength and overcome the weaknesses of the two approaches by applying the doubly robust technique to the problems of policy evaluation and optimization. We prove that this approach yields accurate value estimates when we have either a good (but not necessarily consistent) model of rewards or a good (but not necessarily consistent) model of past policy. Extensive empirical comparison demonstrates that the doubly robust approach uniformly improves over existing techniques, achieving both lower variance in value estimation and better policies. As such, we expect the doubly robust approach to become common practice.

Doubly Robust Policy Evaluation and Learning

TL;DR

The paper tackles offline policy evaluation and learning in contextual bandits with partial rewards. It introduces a doubly robust estimator that combines a reward model with a past-policy model, ensuring unbiasedness if either component is accurate and reducing variance relative to IPS. Through non-asymptotic bias and variance analyses, it shows DR often outperforms DM and IPS, particularly in off-policy evaluation and policy optimization tasks. Empirical results on benchmark multiclass classification datasets and a large-scale real-world dataset demonstrate consistent improvements in estimation accuracy and classifier performance. The work argues that the DR approach should become standard practice in contextual bandit settings due to its robustness and practical benefits.

Abstract

We study decision making in environments where the reward is only partially observed, but can be modeled as a function of an action and an observed context. This setting, known as contextual bandits, encompasses a wide variety of applications including health-care policy and Internet advertising. A central task is evaluation of a new policy given historic data consisting of contexts, actions and received rewards. The key challenge is that the past data typically does not faithfully represent proportions of actions taken by a new policy. Previous approaches rely either on models of rewards or models of the past policy. The former are plagued by a large bias whereas the latter have a large variance. In this work, we leverage the strength and overcome the weaknesses of the two approaches by applying the doubly robust technique to the problems of policy evaluation and optimization. We prove that this approach yields accurate value estimates when we have either a good (but not necessarily consistent) model of rewards or a good (but not necessarily consistent) model of past policy. Extensive empirical comparison demonstrates that the doubly robust approach uniformly improves over existing techniques, achieving both lower variance in value estimation and better policies. As such, we expect the doubly robust approach to become common practice.

Paper Structure

This paper contains 16 sections, 2 theorems, 13 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Prior Work
  3. Problem Definition and Approach
  4. Existing Approaches
  5. Doubly Robust Estimator
  6. Bias Analysis
  7. Variance Analysis
  8. Experiments
  9. Multiclass Classification with Bandit Feedback
  10. Data Setup
  11. Policy Evaluation
  12. Policy Optimization
  13. Estimating Average User Visits
  14. Conclusions
  15. Direct Loss Minimization
  16. ...and 1 more sections

Key Result

Theorem 1

Let $\Delta$ and $\delta$ be defined as above. Then, the bias of the doubly robust estimator is If the past policy and the past policy estimate are stationary (i.e., independent of $h$), the expression simplifies to

Figures (3)

  • Figure 1: Bias (upper) and rmse (lower) of the three estimators for classification error. See Table \ref{['tbl:class-eval']} for precise numbers.
  • Figure 2: Classification error of DLM (upper) and filter tree (lower). Note that the representations used by DLM and the trees differ radically, conflating any comparison between the approaches. However, the Offset and Filter Tree approaches share a similar representation, so differences in performance are purely a matter of superior optimization. See Table \ref{['tbl:class-opt']} for precise numbers.
  • Figure 3: Comparison of IPS and DR: rmse (top), bias (bottom). The ground truth value is $23.8$.

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2