Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Doubly-Robust Off-Policy Evaluation with Estimated Logging Policy

Kyungbok Lee, Myunghee Cho Paik

TL;DR

This work addresses off-policy evaluation when both the logging policy and the value function are unknown by introducing DRUnknown, a doubly-robust estimator that jointly learns the logging-policy model and the value-function model to minimize asymptotic variance. The method estimates the logging policy via maximum likelihood and then optimizes the value-function parameters while incorporating the estimation effect through an auxiliary term, achieving consistency if either model is correct and local efficiency if both are correct. DRUnknown leverages an extended function class with an influence-function-based variance objective, yielding a statistically efficient estimator within the DR family and outperforming existing OPE methods in both contextual bandits and reinforcement learning experiments. Theoretical results establish asymptotic normality and efficiency properties, and empirical results on synthetic CB data and UCI datasets, as well as RL benchmarks, demonstrate substantial finite-sample gains in accuracy and reliability.

Abstract

We introduce a novel doubly-robust (DR) off-policy evaluation (OPE) estimator for Markov decision processes, DRUnknown, designed for situations where both the logging policy and the value function are unknown. The proposed estimator initially estimates the logging policy and then estimates the value function model by minimizing the asymptotic variance of the estimator while considering the estimating effect of the logging policy. When the logging policy model is correctly specified, DRUnknown achieves the smallest asymptotic variance within the class containing existing OPE estimators. When the value function model is also correctly specified, DRUnknown is optimal as its asymptotic variance reaches the semiparametric lower bound. We present experimental results conducted in contextual bandits and reinforcement learning to compare the performance of DRUnknown with that of existing methods.

Doubly-Robust Off-Policy Evaluation with Estimated Logging Policy

TL;DR

This work addresses off-policy evaluation when both the logging policy and the value function are unknown by introducing DRUnknown, a doubly-robust estimator that jointly learns the logging-policy model and the value-function model to minimize asymptotic variance. The method estimates the logging policy via maximum likelihood and then optimizes the value-function parameters while incorporating the estimation effect through an auxiliary term, achieving consistency if either model is correct and local efficiency if both are correct. DRUnknown leverages an extended function class with an influence-function-based variance objective, yielding a statistically efficient estimator within the DR family and outperforming existing OPE methods in both contextual bandits and reinforcement learning experiments. Theoretical results establish asymptotic normality and efficiency properties, and empirical results on synthetic CB data and UCI datasets, as well as RL benchmarks, demonstrate substantial finite-sample gains in accuracy and reliability.

Abstract

We introduce a novel doubly-robust (DR) off-policy evaluation (OPE) estimator for Markov decision processes, DRUnknown, designed for situations where both the logging policy and the value function are unknown. The proposed estimator initially estimates the logging policy and then estimates the value function model by minimizing the asymptotic variance of the estimator while considering the estimating effect of the logging policy. When the logging policy model is correctly specified, DRUnknown achieves the smallest asymptotic variance within the class containing existing OPE estimators. When the value function model is also correctly specified, DRUnknown is optimal as its asymptotic variance reaches the semiparametric lower bound. We present experimental results conducted in contextual bandits and reinforcement learning to compare the performance of DRUnknown with that of existing methods.
Paper Structure (34 sections, 9 theorems, 50 equations, 4 figures, 3 tables)

This paper contains 34 sections, 9 theorems, 50 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Markov Decision Processes
  4. Off-Policy Evaluation Problem
  5. Previous Methods for Off-Policy Evaluation
  6. Standard Methods for Off-Policy Evaluation
  7. Doubly-Robust Off-Policy Evaluation with Unknown Logging Policy
  8. Proposed Method: DRUnknown
  9. DRUnknown for Contextual Bandits
  10. Intuitive Understanding of DRUnknown for CB: extended function class
  11. DRUnknown for Reinforcement Learning
  12. Theoretical Properties
  13. Experimental Results
  14. Contextual Bandits
  15. Simulation Data
  16. ...and 19 more sections

Key Result

Proposition 1

Consider an estimator $\hat{\beta}$ converging to some $\beta^*$ in probability. If the model $\hat{\mu}(\cdot;\phi)$ is correctly specified, the DR OPE estimator is asymptotically linear with influence function $\eta$: where $c(\beta)$ is a vector that solely depends on $\beta$, and

Figures (4)

  • Figure 1: The relative MSE of the estimators on synthetic dataset with respect to the $n$, the number of samples.
  • Figure 2: Boxplot of estimated values from four estimators on simulation data with $N=100$ and $n=10000$. The true target policy value $V^{\pi}$ is indicated by the blue dashed line.
  • Figure 3: The logarithm of the relative MSE from three estimators on the glass dataset as a function of $\alpha$.
  • Figure 4: The CDF of squared errors for 4 estimators in ModelWin.

Theorems & Definitions (15)

  • Proposition 1: Asymptotic Equivalence of $\widehat{V}$ for CB
  • Proposition 2: Variance of $\widetilde{V}^{\text{DR}}$
  • Proposition 3: Doubly-Robustness
  • Proposition 4: Asymptotic Equivalence of $\widehat{V}$ for RL
  • Proposition 5: Variance of $\widetilde{V}$ for RL
  • Theorem 6: Asymptotic distribution
  • Proposition 7: Local Efficiency
  • Proposition 8: Intrinsic Efficiency
  • Corollary 9: Comparison to Existing Algorithms
  • proof
  • ...and 5 more