Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning

Ye Shen, Hengrui Cai, Rui Song

TL;DR

This work tackles online policy evaluation under contextual bandits, where the optimal policy is unknown and data are influenced by exploration-exploitation. It introduces DREAM, a doubly robust interval estimator that accounts for the probability of exploration $\kappa_t(\boldsymbol{x}_t)$ and yields Wald-type confidence intervals for the optimal-value $V^*$. The authors establish theoretical guarantees: explicit bounds on exploration probability for UCB, TS, and EG; asymptotic normality of online ridge estimators; and a double-robust asymptotic distribution for DREAM with a consistent variance estimator. Through simulations and OpenML-based real data experiments, DREAM demonstrates robust coverage, efficiency, and practical utility for real-time policy evaluation and potential early stopping in online experiments.

Abstract

Evaluating the performance of an ongoing policy plays a vital role in many areas such as medicine and economics, to provide crucial instructions on the early-stop of the online experiment and timely feedback from the environment. Policy evaluation in online learning thus attracts increasing attention by inferring the mean outcome of the optimal policy (i.e., the value) in real-time. Yet, such a problem is particularly challenging due to the dependent data generated in the online environment, the unknown optimal policy, and the complex exploration and exploitation trade-off in the adaptive experiment. In this paper, we aim to overcome these difficulties in policy evaluation for online learning. We explicitly derive the probability of exploration that quantifies the probability of exploring non-optimal actions under commonly used bandit algorithms. We use this probability to conduct valid inference on the online conditional mean estimator under each action and develop the doubly robust interval estimation (DREAM) method to infer the value under the estimated optimal policy in online learning. The proposed value estimator provides double protection for consistency and is asymptotically normal with a Wald-type confidence interval provided. Extensive simulation studies and real data applications are conducted to demonstrate the empirical validity of the proposed DREAM method.

Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning

TL;DR

This work tackles online policy evaluation under contextual bandits, where the optimal policy is unknown and data are influenced by exploration-exploitation. It introduces DREAM, a doubly robust interval estimator that accounts for the probability of exploration and yields Wald-type confidence intervals for the optimal-value . The authors establish theoretical guarantees: explicit bounds on exploration probability for UCB, TS, and EG; asymptotic normality of online ridge estimators; and a double-robust asymptotic distribution for DREAM with a consistent variance estimator. Through simulations and OpenML-based real data experiments, DREAM demonstrates robust coverage, efficiency, and practical utility for real-time policy evaluation and potential early stopping in online experiments.

Abstract

Evaluating the performance of an ongoing policy plays a vital role in many areas such as medicine and economics, to provide crucial instructions on the early-stop of the online experiment and timely feedback from the environment. Policy evaluation in online learning thus attracts increasing attention by inferring the mean outcome of the optimal policy (i.e., the value) in real-time. Yet, such a problem is particularly challenging due to the dependent data generated in the online environment, the unknown optimal policy, and the complex exploration and exploitation trade-off in the adaptive experiment. In this paper, we aim to overcome these difficulties in policy evaluation for online learning. We explicitly derive the probability of exploration that quantifies the probability of exploring non-optimal actions under commonly used bandit algorithms. We use this probability to conduct valid inference on the online conditional mean estimator under each action and develop the doubly robust interval estimation (DREAM) method to infer the value under the estimated optimal policy in online learning. The proposed value estimator provides double protection for consistency and is asymptotically normal with a Wald-type confidence interval provided. Extensive simulation studies and real data applications are conducted to demonstrate the empirical validity of the proposed DREAM method.

Paper Structure

This paper contains 27 sections, 9 theorems, 257 equations, 9 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works and Challenges
  3. Our Contributions
  4. Problem Formulation
  5. Framework
  6. Bandit Algorithms
  7. Probability of Exploration
  8. Doubly Robust Interval Estimation
  9. Theoretical Results
  10. Bounding the Probability of Exploration
  11. Asymptotic Normality of Online Ridge Estimator
  12. Asymptotic Normality and Robustness for DREAM
  13. Simulation Studies
  14. Real Data Application
  15. Discussion
  16. ...and 12 more sections

Key Result

Lemma 4.1

(Tail bound for the online ridge estimator). In the online contextual bandit under UCB, TS, or EG, with Assumptions ALx and Clipping hold, we have that for any $h>0$, the probability of the online ridge estimator bounded within its true as

Figures (9)

  • Figure 1: Left panel: the architecture of offline policy evaluation, with offline context-action-outcome triples $\{({\boldsymbol{x}_t},a_t,r_t)\}$ stored in the buffer to learn the value under a target policy $\pi^*$ with data generated by a behavior policy $\pi_b$. Right panel: the architecture of doubly robust interval estimation (DREAM) method for policy evaluation in online learning, where the context-action-outcome triple at time $t$, $({\boldsymbol{x}_t},a_t,r_t)$, is stored in the buffer to update the bandit policy $\pi_t$ and in the meantime to evaluate its performance.
  • Figure 2: Results by DREAM with UCB under different model specifications in comparison to the averaged reward. Left panel: the coverage probabilities of the 95% two-sided Wald-type confidence interval, with the red line representing the nominal level at 95%. Middle panel: the bias between the estimated value and the true value. Right panel: the ratio between the standard error and the Monte Carlo standard deviation, with the red line representing the nominal level at 1.
  • Figure 3: Results by DREAM with TS under different model specifications in comparison to the averaged reward. Left panel: the coverage probabilities of the 95% two-sided Wald-type confidence interval, with the red line representing the nominal level at 95%. Middle panel: the bias between the estimated value and the true value. Right panel: the ratio between the standard error and the Monte Carlo standard deviation, with the red line representing the nominal level at 1.
  • Figure 4: Results by DREAM with EG under different model specifications in comparison to the averaged reward. Left panel: the coverage probabilities of the 95% two-sided Wald-type confidence interval, with the red line representing the nominal level at 95%. Middle panel: the bias between the estimated value and the true value. Right panel: the ratio between the standard error and the Monte Carlo standard deviation, with the red line representing the nominal level at 1.
  • Figure 5: The coverage probabilities of the 95% two-sided Wald-type confidence interval by the proposed value estimator under DREAM in comparison to the averaged reward, under different contextual bandit algorithms. Left panel: the results for the SEA50 dataset. Right panel: the results for the SEA50000 dataset. The red line represents the nominal level at 95%.
  • ...and 4 more figures

Theorems & Definitions (9)

  • Lemma 4.1
  • Corollary 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Corollary 2
  • Lemma B.1
  • Lemma B.2
  • Lemma B.3