Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning

Otmane Sakhi, Imad Aouali, Pierre Alquier, Nicolas Chopin

TL;DR

This work introduces novel, fully empirical concentration bounds for a broad class of importance weighting risk estimators, and motivates a novel estimator, that logarithmically smooths large importance weights, that results in improved policy selection and learning strategies.

Abstract

This work investigates the offline formulation of the contextual bandit problem, where the goal is to leverage past interactions collected under a behavior policy to evaluate, select, and learn new, potentially better-performing, policies. Motivated by critical applications, we move beyond point estimators. Instead, we adopt the principle of pessimism where we construct upper bounds that assess a policy's worst-case performance, enabling us to confidently select and learn improved policies. Precisely, we introduce novel, fully empirical concentration bounds for a broad class of importance weighting risk estimators. These bounds are general enough to cover most existing estimators and pave the way for the development of new ones. In particular, our pursuit of the tightest bound within this class motivates a novel estimator (LS), that logarithmically smooths large importance weights. The bound for LS is provably tighter than its competitors, and naturally results in improved policy selection and learning strategies. Extensive policy evaluation, selection, and learning experiments highlight the versatility and favorable performance of LS.

Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning

TL;DR

This work introduces novel, fully empirical concentration bounds for a broad class of importance weighting risk estimators, and motivates a novel estimator, that logarithmically smooths large importance weights, that results in improved policy selection and learning strategies.

Abstract

This work investigates the offline formulation of the contextual bandit problem, where the goal is to leverage past interactions collected under a behavior policy to evaluate, select, and learn new, potentially better-performing, policies. Motivated by critical applications, we move beyond point estimators. Instead, we adopt the principle of pessimism where we construct upper bounds that assess a policy's worst-case performance, enabling us to confidently select and learn improved policies. Precisely, we introduce novel, fully empirical concentration bounds for a broad class of importance weighting risk estimators. These bounds are general enough to cover most existing estimators and pave the way for the development of new ones. In particular, our pursuit of the tightest bound within this class motivates a novel estimator (LS), that logarithmically smooths large importance weights. The bound for LS is provably tighter than its competitors, and naturally results in improved policy selection and learning strategies. Extensive policy evaluation, selection, and learning experiments highlight the versatility and favorable performance of LS.
Paper Structure (57 sections, 34 theorems, 193 equations, 6 figures, 7 tables)

This paper contains 57 sections, 34 theorems, 193 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Setting and background
  3. Pessimistic off-policy evaluation
  4. Preliminaries and unified risk bounds
  5. Global clipping
  6. Logarithmic smoothing
  7. Off-policy selection and learning
  8. Off-policy selection
  9. Off-policy learning
  10. Experiments
  11. Off-policy evaluation and selection experiments
  12. Off-policy learning experiments
  13. Conclusion
  14. Limitations
  15. Broader impact
  16. ...and 42 more sections

Key Result

Proposition 1

Let $\pi \in \Pi$, $L \ge 1$, $\delta \in (0,1]$, $\lambda > 0$, and $h$ satisfying eq:condition. Then it holds with probability at least $1 - \delta$ that where $\psi_\lambda$ and $\hat{\mathcal{M}}^{h, \ell}_n(\pi)$ are both defined in eq:kth-moment, and recall that $\psi_\lambda(x) \leq x$.

Figures (6)

  • Figure 1: $\texttt{LS}$ with different $\lambda$s.
  • Figure 2: Results for OPE and OPS experiments.
  • Figure 3: \ref{['ocb_high_order']} for different values of $L$ and with different regularized IPS $h$.
  • Figure 4: Comparison of Logarithmic Smoothing and Clipping.
  • Figure 5: OPL: Guaranteed Risk given by the different bounds. We observe that our LS-LIN dominates all other bounds. IX comes close, especially on EMNIST and nuswide
  • ...and 1 more figures

Theorems & Definitions (50)

  • Proposition 1: Empirical moments risk bound
  • Proposition 2: Impact of $L$
  • Proposition 3: Comparison of our bounds
  • Corollary 4: Empirical second-moment risk bound with $L=1$
  • Corollary 5: Empirical infinite-moment bound with $L \to \infty$
  • Proposition 6: Bias-variance trade-off
  • Proposition 7: Sub-Gaussianity and comparison with metelli2021subgaussian
  • Proposition 8: Comparison with IX of gabbianelli2023importance
  • Proposition 9: Suboptimality of our selection strategy in \ref{['eq:selection_strategy']}
  • Proposition 10: PAC-Bayes learning bound for $\hat{R}^{\lambda{\textsc{-lin}}}_n$
  • ...and 40 more