Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Refined PAC-Bayes Bounds for Offline Bandits

Amaury Gouverneur, Tobias J. Oechtering, Mikael Skoglund

TL;DR

This work tackles off-policy evaluation in bandit settings by refining PAC-Bayes bounds for the importance-sampling reward estimator $\widehat{\mathscr{R}}^{\mathrm{IS}}(\pi,H^t)$. Building on prior PAC-Bayes results and a parameter-optimization method based on discretizing event spaces, it introduces two parameter-free bounds, one using Hoeffding-Azuma and one using Bernstein inequalities, that achieve near-optimal rates. The key technical contribution is an optimized, data-adaptive bound construction that removes the need to pre-specify the trade-off parameter $\lambda$, while preserving uniform validity over policies. These refined bounds improve reliability of offline policy evaluation and lay groundwork for PAC-Bayes regret analyses in bandit problems, with potential impact on offline policy selection and safe exploration in sequential decision tasks.

Abstract

In this paper, we present refined probabilistic bounds on empirical reward estimates for off-policy learning in bandit problems. We build on the PAC-Bayesian bounds from Seldin et al. (2010) and improve on their results using a new parameter optimization approach introduced by Rodríguez et al. (2024). This technique is based on a discretization of the space of possible events to optimize the "in probability" parameter. We provide two parameter-free PAC-Bayes bounds, one based on Hoeffding-Azuma's inequality and the other based on Bernstein's inequality. We prove that our bounds are almost optimal as they recover the same rate as would be obtained by setting the "in probability" parameter after the realization of the data.

Refined PAC-Bayes Bounds for Offline Bandits

TL;DR

This work tackles off-policy evaluation in bandit settings by refining PAC-Bayes bounds for the importance-sampling reward estimator . Building on prior PAC-Bayes results and a parameter-optimization method based on discretizing event spaces, it introduces two parameter-free bounds, one using Hoeffding-Azuma and one using Bernstein inequalities, that achieve near-optimal rates. The key technical contribution is an optimized, data-adaptive bound construction that removes the need to pre-specify the trade-off parameter , while preserving uniform validity over policies. These refined bounds improve reliability of offline policy evaluation and lay groundwork for PAC-Bayes regret analyses in bandit problems, with potential impact on offline policy selection and safe exploration in sequential decision tasks.

Abstract

In this paper, we present refined probabilistic bounds on empirical reward estimates for off-policy learning in bandit problems. We build on the PAC-Bayesian bounds from Seldin et al. (2010) and improve on their results using a new parameter optimization approach introduced by Rodríguez et al. (2024). This technique is based on a discretization of the space of possible events to optimize the "in probability" parameter. We provide two parameter-free PAC-Bayes bounds, one based on Hoeffding-Azuma's inequality and the other based on Bernstein's inequality. We prove that our bounds are almost optimal as they recover the same rate as would be obtained by setting the "in probability" parameter after the realization of the data.

Paper Structure

This paper contains 11 sections, 7 theorems, 26 equations.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. General Notation
  4. Multi-Armed Bandits
  5. Contextual Bandits
  6. Online and Offline Settings
  7. PAC-Bayes Bounds
  8. Bandit Problems and Martingales
  9. PAC-Bayes Bound for $\widehat{\mathscr{R}}^{\mathrm{IS}}(\pi,H^t)$
  10. Parameter-free PAC-Bayes bound for $\widehat{\mathscr{R}}^{\mathrm{IS}}(\pi,H^t)$
  11. Conclusion

Key Result

Lemma 1

For any measurable, bounded function $h: \mathcal{A} \to \mathbb{R}$ and any probability distribution on $\mathcal{A}$, $\mu \in \Pi$, such that $\mathbb{E}_{A\sim \mu}[e^{h(A)}] < \infty$, we have

Theorems & Definitions (10)

  • Example 1: Drug dosage
  • Example 2: Online advertising
  • Definition 1: Martingale
  • Lemma 1: Donsker-Varadhan's variational formula
  • Lemma 2: Hoeffding-Azuma's inequality
  • Lemma 3: Bernstein's inequality
  • Proposition 1: Hoeffding PAC-Bayes bound for $\hat{r}^{\mathrm{IS}}$
  • Proposition 2: Bernstein PAC-Bayes bound for $\hat{r}^{\mathrm{IS}}$
  • Theorem 1: Optimized Hoeffding PAC-Bayes bound
  • Theorem 2: Optimized Bernstein PAC-Bayes bound