Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empirical Bayesian Multi-Bandit Learning

Xia Jiang, Rong J. B. Zhu

TL;DR

This article proposes a novel hierarchical Bayesian framework for learning in various bandit instances that captures both the heterogeneity and the correlations among different bandit instances through a hierarchical Bayesian model, enabling effective information sharing while accommodating instance-specific variations.

Abstract

Multi-task learning in contextual bandits has attracted significant research interest due to its potential to enhance decision-making across multiple related tasks by leveraging shared structures and task-specific heterogeneity. In this article, we propose a novel hierarchical Bayesian framework for learning in various bandit instances. This framework captures both the heterogeneity and the correlations among different bandit instances through a hierarchical Bayesian model, enabling effective information sharing while accommodating instance-specific variations. Unlike previous methods that overlook the learning of the covariance structure across bandits, we introduce an empirical Bayesian approach to estimate the covariance matrix of the prior distribution. This enhances both the practicality and flexibility of learning across multi-bandits. Building on this approach, we develop two efficient algorithms: ebmTS (Empirical Bayesian Multi-Bandit Thompson Sampling) and ebmUCB (Empirical Bayesian Multi-Bandit Upper Confidence Bound), both of which incorporate the estimated prior into the decision-making process. We provide the frequentist regret upper bounds for the proposed algorithms, thereby filling a research gap in the field of multi-bandit problems. Extensive experiments on both synthetic and real-world datasets demonstrate the superior performance of our algorithms, particularly in complex environments. Our methods achieve lower cumulative regret compared to existing techniques, highlighting their effectiveness in balancing exploration and exploitation across multi-bandits.

Empirical Bayesian Multi-Bandit Learning

TL;DR

This article proposes a novel hierarchical Bayesian framework for learning in various bandit instances that captures both the heterogeneity and the correlations among different bandit instances through a hierarchical Bayesian model, enabling effective information sharing while accommodating instance-specific variations.

Abstract

Multi-task learning in contextual bandits has attracted significant research interest due to its potential to enhance decision-making across multiple related tasks by leveraging shared structures and task-specific heterogeneity. In this article, we propose a novel hierarchical Bayesian framework for learning in various bandit instances. This framework captures both the heterogeneity and the correlations among different bandit instances through a hierarchical Bayesian model, enabling effective information sharing while accommodating instance-specific variations. Unlike previous methods that overlook the learning of the covariance structure across bandits, we introduce an empirical Bayesian approach to estimate the covariance matrix of the prior distribution. This enhances both the practicality and flexibility of learning across multi-bandits. Building on this approach, we develop two efficient algorithms: ebmTS (Empirical Bayesian Multi-Bandit Thompson Sampling) and ebmUCB (Empirical Bayesian Multi-Bandit Upper Confidence Bound), both of which incorporate the estimated prior into the decision-making process. We provide the frequentist regret upper bounds for the proposed algorithms, thereby filling a research gap in the field of multi-bandit problems. Extensive experiments on both synthetic and real-world datasets demonstrate the superior performance of our algorithms, particularly in complex environments. Our methods achieve lower cumulative regret compared to existing techniques, highlighting their effectiveness in balancing exploration and exploitation across multi-bandits.

Paper Structure

This paper contains 32 sections, 17 theorems, 110 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation
  4. Estimation
  5. Estimation of beta_k,j and its Uncertainty
  6. Step 1: Conditional Expectation of $\boldsymbol{\beta}_{k,j}$
  7. Step 2: Estimation of $\boldsymbol{\beta}_{k0}$
  8. Step 3: Estimation of $\boldsymbol{\beta}_{k, j}$
  9. Estimation error bound analysis
  10. Prediction of mu_k,j,t under Context x_t and its Uncertainty
  11. Estimation of Sigma_k and sigma_k^2
  12. Algorithms
  13. ebmTS
  14. ebmUCB
  15. Regret Analysis
  16. ...and 17 more sections

Key Result

Theorem 4.2

Under the Assumption assumption, the estimator $\widehat{\boldsymbol{\beta}}_{k,j,t}$ that incorporates prior information satisfies the following inequality with probability at least $1-\delta$, for any fixed $t\geq 1$ and $\mathbf{x} \in \mathbb{R}^d$: where $m_1=\max\{\lambda_1/\lambda, 1\}$, $m_2=\max\{\lambda, \sigma^2\lambda_1\}$, and $m_3=c_{\beta}\sqrt{\lambda_1}$.

Figures (5)

  • Figure 1: Performance under $N=10$, $K=5$, $d=3$.
  • Figure 2: Performance under Sparse heterogeneity.
  • Figure 3: Performance under different context distributions, $N$, $K$, and $d$.
  • Figure 4: Performance of real-world datasets.
  • Figure 5: Performance under $N=10$, $K=5$, $d=3$.

Theorems & Definitions (17)

  • Theorem 4.2
  • Lemma 4.3
  • Theorem 6.1
  • Theorem 6.2
  • Lemma A.1
  • Lemma A.2
  • Lemma C.1
  • Lemma C.2
  • Lemma C.3
  • Lemma D.1
  • ...and 7 more