Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Baseline Corrections for Off-Policy Contextual Bandits

Shashank Gupta, Olivier Jeunen, Harrie Oosterhuis, Maarten de Rijke

TL;DR

This work addresses unbiased off-policy learning and evaluation for contextual bandits by unifying existing control variate techniques under a single baseline-correction framework. It derives a variance-minimizing baseline for gradient estimates and a closed-form optimal baseline for value estimation, providing practical, unbiased estimators with lower variance and data requirements. The proposed beta-IPS method outperforms IPS, SNIPS, and DR across learning and evaluation tasks, both in simulation with OBP and on real-world logs, demonstrating faster convergence and more accurate policy value estimates. Overall, the approach offers a principled, parameter-free route to improved offline policy learning and evaluation in large-scale recommendation and ranking settings.

Abstract

The off-policy learning paradigm allows for recommender systems and general ranking applications to be framed as decision-making problems, where we aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. With unbiasedness comes potentially high variance, and prevalent methods exist to reduce estimation variance. These methods typically make use of control variates, either additive (i.e., baseline corrections or doubly robust methods) or multiplicative (i.e., self-normalisation). Our work unifies these approaches by proposing a single framework built on their equivalence in learning scenarios. The foundation of our framework is the derivation of an equivalent baseline correction for all of the existing control variates. Consequently, our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it. This optimal estimator brings significantly improved performance in both evaluation and learning, and minimizes data requirements. Empirical observations corroborate our theoretical findings.

Optimal Baseline Corrections for Off-Policy Contextual Bandits

TL;DR

This work addresses unbiased off-policy learning and evaluation for contextual bandits by unifying existing control variate techniques under a single baseline-correction framework. It derives a variance-minimizing baseline for gradient estimates and a closed-form optimal baseline for value estimation, providing practical, unbiased estimators with lower variance and data requirements. The proposed beta-IPS method outperforms IPS, SNIPS, and DR across learning and evaluation tasks, both in simulation with OBP and on real-world logs, demonstrating faster convergence and more accurate policy value estimates. Overall, the approach offers a principled, parameter-free route to improved offline policy learning and evaluation in large-scale recommendation and ranking settings.

Abstract

The off-policy learning paradigm allows for recommender systems and general ranking applications to be framed as decision-making problems, where we aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. With unbiasedness comes potentially high variance, and prevalent methods exist to reduce estimation variance. These methods typically make use of control variates, either additive (i.e., baseline corrections or doubly robust methods) or multiplicative (i.e., self-normalisation). Our work unifies these approaches by proposing a single framework built on their equivalence in learning scenarios. The foundation of our framework is the derivation of an equivalent baseline correction for all of the existing control variates. Consequently, our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it. This optimal estimator brings significantly improved performance in both evaluation and learning, and minimizes data requirements. Empirical observations corroborate our theoretical findings.
Paper Structure (14 sections, 2 theorems, 31 equations, 5 figures, 1 table)

This paper contains 14 sections, 2 theorems, 31 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction & Motivation
  2. Background and Related Work
  3. On-policy contextual bandits
  4. Off-policy estimation for general bandits
  5. Unifying Off-Policy Estimators
  6. A unified off-policy estimator
  7. Minimizing gradient variance
  8. Minimizing estimation variance
  9. Experimental Setup
  10. Results and Discussion
  11. Off-policy learning performance (RQ1--3)
  12. Off-policy evaluation performance (RQ4)
  13. Conclusion and Future Work
  14. Appendix: Off-policy Estimator Variance

Key Result

Theorem 1

Within the family of gradient estimators with a global additive control variate, i.e., $\beta$-IPS (Eq. eq:MC_gradient_bIPS), IPS (Eq. eq:MC_gradient_IPS), BanditNet (Eq. eq:banditnet_grad), and DR with a constant correction (Eq. eq:dr_grad_const_rew), $\beta$-IPS with our proposed choice of $\beta$

Figures (5)

  • Figure 1: Performance of different off-policy learning methods trained in a full-batch gradient descent fashion in terms of the policy value on the test set. x-axis corresponds to the training epoch during the optimization (we use a maximum of 500 epochs for all methods), and y-axis corresponds to the policy value. A decaying learning rate is used. Reported results are averages over 32 independent runs with 95% confidence interval.
  • Figure 2: Performance of different off-policy learning methods trained in a mini-batch gradient descent fashion in terms of policy value on the test set. The axis labels are similar to Figure \ref{['fig:full_batch_val']}.
  • Figure 3: Empirical variance of the gradient of different off-policy learning estimators in a mini-batch optimization setup with varying learning rates (in title). We compute gradient variance for each mini-batch during training and then report the average value across all mini-batches in a training epoch. Results are averaged across 32 independent runs with 95% confidence interval.
  • Figure 4: Mean Squared Error (MSE) of different off-policy estimators with varying action space (from left to right), and varying inverse temperature parameter of the softmax logging policy (from top to bottom). X-axis corresponds to the size of the logged data simulated (ranging from $10^2$ to $10^6$), and the y-axis corresponds to the MSE (evaluated over 100 independent samples of the synthetic data) along with 95% confidence interval. Each row corresponds to a different setting of inverse temperature of the softmax logging policy. We only consider unbiased (asymptotically or otherwise) estimators.
  • Figure 5: Empirical variance of different off-policy estimators with varying action space (from left to right), and varying sub-optimality of a temperature-based softmax behavior policy (from top to bottom). The x-axis corresponds to the size of the logged data simulated (ranging from $10^2$ to $10^6$), and the y-axis corresponds to the variance of different estimators (evaluated over 100 independent samples of the synthetic data) along with 95% confidence interval. Each row corresponds to a different optimality level of the logging policy, decided by the inverse temperature parameter. We only consider unbiased (asymptotically or otherwise) estimators.

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2