Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Thompson Sampling via Information Relaxation for Budgeted Multi-armed Bandits

Woojin Jeong, Seungki Min

TL;DR

This work tackles budgeted multi-armed bandits where each arm incurs a known cost and the total resource budget is limited. It extends Thompson Sampling via the Information Relaxation Sampling (IRS) framework to account for remaining budget, producing algorithms like IRS.FH and IRS.V-Zero that simulate future belief dynamics and optimize decisions accordingly; these policies interpolate between BTS and the Bayes-optimal policy. The authors establish monotone, tighter performance bounds such as $W^{\text{BTS}} \ge W^{\text{IRS.FH}} \ge W^{\text{IRS.V-Zero}} \ge V^{\star}$ and derive suboptimality bounds under natural exponential-family assumptions, demonstrating incremental improvements over BTS both theoretically and empirically. They further extend IRS to random-cost settings with simple and penalty-based extensions, achieving substantial regret reductions in Beta-Bernoulli and real online-ad budgets, indicating practical significance for budget-constrained sequential decision tasks. Overall, IRS provides a principled, tunable approach to budget-aware exploration with provable guarantees and strong empirical performance gains.

Abstract

We consider a Bayesian budgeted multi-armed bandit problem, in which each arm consumes a different amount of resources when selected and there is a budget constraint on the total amount of resources that can be used. Budgeted Thompson Sampling (BTS) offers a very effective heuristic to this problem, but its arm-selection rule does not take into account the remaining budget information. We adopt \textit{Information Relaxation Sampling} framework that generalizes Thompson Sampling for classical $K$-armed bandit problems, and propose a series of algorithms that are randomized like BTS but more carefully optimize their decisions with respect to the budget constraint. In a one-to-one correspondence with these algorithms, a series of performance benchmarks that improve the conventional benchmark are also suggested. Our theoretical analysis and simulation results show that our algorithms (and our benchmarks) make incremental improvements over BTS (respectively, the conventional benchmark) across various settings including a real-world example.

Improving Thompson Sampling via Information Relaxation for Budgeted Multi-armed Bandits

TL;DR

This work tackles budgeted multi-armed bandits where each arm incurs a known cost and the total resource budget is limited. It extends Thompson Sampling via the Information Relaxation Sampling (IRS) framework to account for remaining budget, producing algorithms like IRS.FH and IRS.V-Zero that simulate future belief dynamics and optimize decisions accordingly; these policies interpolate between BTS and the Bayes-optimal policy. The authors establish monotone, tighter performance bounds such as and derive suboptimality bounds under natural exponential-family assumptions, demonstrating incremental improvements over BTS both theoretically and empirically. They further extend IRS to random-cost settings with simple and penalty-based extensions, achieving substantial regret reductions in Beta-Bernoulli and real online-ad budgets, indicating practical significance for budget-constrained sequential decision tasks. Overall, IRS provides a principled, tunable approach to budget-aware exploration with provable guarantees and strong empirical performance gains.

Abstract

We consider a Bayesian budgeted multi-armed bandit problem, in which each arm consumes a different amount of resources when selected and there is a budget constraint on the total amount of resources that can be used. Budgeted Thompson Sampling (BTS) offers a very effective heuristic to this problem, but its arm-selection rule does not take into account the remaining budget information. We adopt \textit{Information Relaxation Sampling} framework that generalizes Thompson Sampling for classical -armed bandit problems, and propose a series of algorithms that are randomized like BTS but more carefully optimize their decisions with respect to the budget constraint. In a one-to-one correspondence with these algorithms, a series of performance benchmarks that improve the conventional benchmark are also suggested. Our theoretical analysis and simulation results show that our algorithms (and our benchmarks) make incremental improvements over BTS (respectively, the conventional benchmark) across various settings including a real-world example.
Paper Structure (50 sections, 5 theorems, 126 equations, 2 figures, 11 algorithms)

This paper contains 50 sections, 5 theorems, 126 equations, 2 figures, 11 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation and Preliminaries
  3. Rewards and costs.
  4. Bayesian framework.
  5. Mean reward estimates.
  6. Policy and performance.
  7. Performance bound and regret.
  8. Bayesian optimal policy.
  9. Algorithms
  10. Budgeted Thompson Sampling
  11. IRS.FH
  12. Policy $\pi^\text{IRS.FH}$.
  13. Bound $W^\text{IRS.FH}$.
  14. IRS.V-Zero
  15. Policy $\pi^\text{IRS.V-Zero}.$
  16. ...and 35 more sections

Key Result

Theorem 1

For any Bayesian budgeted MAB, we have

Figures (2)

  • Figure 1: From left to right, simulation results in (a) the Beta-Bernoulli MAB with two arms, (b) the Beta-Bernoulli MAB with five arms, and (c) the Beta-Binomial MAB as a real-world example arising in online advertisement business.
  • Figure 2: Simulation result in the Beta-Bernoulli MAB with two arms incurring random costs.

Theorems & Definitions (12)

  • Remark 1
  • Theorem 1: Monotonicity of performance bounds
  • Theorem 2: Suboptimality gap
  • Definition 1: Dual feasible penalty function, adopted from min2019thompson
  • Theorem 3: Weak duality and strong duality, adopted from min2019thompson
  • Remark 2
  • Remark 3
  • Lemma 1: min2019thompson
  • Definition 2: Supersolution
  • Remark 4
  • ...and 2 more