Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian Regret Minimization in Offline Bandits

Marek Petrik, Guy Tennenholtz, Mohammad Ghavamzadeh

TL;DR

This work addresses minimizing Bayesian regret in offline linear bandits by moving away from traditional lower confidence bound (LCB) strategies and instead directly minimizing upper bounds on Bayesian regret via VaR reformulation. The authors derive two complementary analytic bounds under Gaussian and sub-Gaussian posteriors and design BRMOB, a conic-optimization–based algorithm that returns randomized policies to hedge regret. They prove a matching lower bound, establishing tightness of the proposed bounds, and show BRMOB outperforms LCB-based methods on synthetic domains, especially when posterior uncertainty is large. The results challenge the efficacy of LCB for Bayesian regret minimization and point to regret-bound–driven optimization as a promising direction, with extensions to sub-Gaussian posteriors and contextual settings discussed for future work.

Abstract

We study how to make decisions that minimize Bayesian regret in offline linear bandits. Prior work suggests that one must take actions with maximum lower confidence bound (LCB) on their reward. We argue that the reliance on LCB is inherently flawed in this setting and propose a new algorithm that directly minimizes upper bounds on the Bayesian regret using efficient conic optimization solvers. Our bounds build heavily on new connections to monetary risk measures. Proving a matching lower bound, we show that our upper bounds are tight, and by minimizing them we are guaranteed to outperform the LCB approach. Our numerical results on synthetic domains confirm that our approach is superior to LCB.

Bayesian Regret Minimization in Offline Bandits

TL;DR

This work addresses minimizing Bayesian regret in offline linear bandits by moving away from traditional lower confidence bound (LCB) strategies and instead directly minimizing upper bounds on Bayesian regret via VaR reformulation. The authors derive two complementary analytic bounds under Gaussian and sub-Gaussian posteriors and design BRMOB, a conic-optimization–based algorithm that returns randomized policies to hedge regret. They prove a matching lower bound, establishing tightness of the proposed bounds, and show BRMOB outperforms LCB-based methods on synthetic domains, especially when posterior uncertainty is large. The results challenge the efficacy of LCB for Bayesian regret minimization and point to regret-bound–driven optimization as a promising direction, with extensions to sub-Gaussian posteriors and contextual settings discussed for future work.

Abstract

We study how to make decisions that minimize Bayesian regret in offline linear bandits. Prior work suggests that one must take actions with maximum lower confidence bound (LCB) on their reward. We argue that the reliance on LCB is inherently flawed in this setting and propose a new algorithm that directly minimizes upper bounds on the Bayesian regret using efficient conic optimization solvers. Our bounds build heavily on new connections to monetary risk measures. Proving a matching lower bound, we show that our upper bounds are tight, and by minimizing them we are guaranteed to outperform the LCB approach. Our numerical results on synthetic domains confirm that our approach is superior to LCB.
Paper Structure (30 sections, 17 theorems, 85 equations, 8 figures)

This paper contains 30 sections, 17 theorems, 85 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Bayesian Offline Bandits
  4. Problem Definition
  5. Baseline Algorithms
  6. Lower Confidence Bound (LCB)
  7. Scenario-based Methods
  8. Minimizing Analytical Regret Bounds
  9. Bayesian Regret Bounds
  10. Optimization Algorithm
  11. Regret Analysis
  12. Sample-Based Regret Bound
  13. Comparison with FlatOPO
  14. Limitation of LCB
  15. Numerical Results
  16. ...and 15 more sections

Key Result

Lemma 4.2

Suppose that $\bm{\tilde{\theta}}_D \sim \mathcal{N}(\bm{\mu}, \bm{\Sigma})$. Then, for any policy $\bm{\pi}\in \Delta_{k}$, the Bayesian regret in eq:regret-high-confidence-VaR can be written as where $\;\tilde{x}_a^{\bm{\pi}}\sim\mathcal{N}(\mu_a^{\bm{\pi}},\sigma_a^{\bm{\pi}})\;$ with

Figures (8)

  • Figure 1: The quotient of the upper bound coefficient $\kappa_{\mathrm{u}}(k)$ and the lower bound coefficient $\kappa_{\mathrm{l}}(k)$.
  • Figure 2: The value of $\beta$ used by FlatOPO in \ref{['exm:lcb-counterexample']}, $\beta_{\mathrm{OPO}}$, and the upper bound $\beta^{\star}$ that may avoid the under-performance of LCB, defined in \ref{['eq:beta-values']}, as functions of the number of actions $k$.
  • Figure 3: Bayesian regret with $k=d=5$(left), $k=d=50$(middle and right). The prior mean is $\bm{\mu}_0 = \bm{0}$(left and middle) and $(\bm{\mu}_0)_a = \sqrt{a}\;$ for $a = 1, \dots 50$(right).
  • Figure 4: Bayesian regret with $d=4$ and $k=10$(left), $k=50$(middle), and $k=100$(right).
  • Figure 5: Bayesian regret with $k=d=5$(left), $k=d=50$(middle and right). The prior mean is $\bm{\mu}_0 = \bm{0}$(left and middle) and $(\bm{\mu}_0)_a = \sqrt{a}\;$ for $a = 1, \dots 50$(right).
  • ...and 3 more figures

Theorems & Definitions (35)

  • Lemma 4.2
  • Theorem 4.3
  • Theorem 4.4
  • Theorem 4.5
  • Proposition 4.6
  • Theorem 5.2
  • Example 1
  • Theorem 5.3
  • Lemma A.1
  • proof
  • ...and 25 more