Bayesian Online Model Selection

TL;DR

This work proves an oracle-style guarantee of an oracle-style guarantee of O( d^* M + T) on the Bayesian regret, where M is the number of base learners, d^* is the regret coefficient of the optimal base learner, and T is the time horizon.

Abstract

Online model selection in Bayesian bandits raises a fundamental exploration challenge: When an environment instance is sampled from a prior distribution, how can we design an adaptive strategy that explores multiple bandit learners and competes with the best one in hindsight? We address this problem by introducing a new Bayesian algorithm for online model selection in stochastic bandits. We prove an oracle-style guarantee of $O\left( d^* M \sqrt{T} + \sqrt{(MT)} \right)$ on the Bayesian regret, where $M$ is the number of base learners, $d^*$ is the regret coefficient of the optimal base learner, and $T$ is the time horizon. We also validate our method empirically across a range of stochastic bandit settings, demonstrating performance that is competitive with the best base learner. Additionally, we study the effect of sharing data among base learners and its role in mitigating prior mis-specification.

Figures (7)

• Figure 1: Empirical Bayesian regret for B-MS vs. TS base learners with different prior specifications ($T=5\times10^3$, $R=500$, $K=2$, $M=4$).
• Figure 2: B-MS vs. UCB base learners with confidence radius $c$ ($T= 10^3, R=100, K=5, M=6$).
• Figure 3: B-MS vs. LinTS base learners with confidence radius $c$ ($T= 1.5\times 10^4, R=100, K=1000, M=5, d=10$).
• Figure 4: B-MS vs. base learners (TS and ILS algorithm) for $T=10^3$, $R=60$, $K=8$, $M=3$.
• Figure 5: Empirical Bayesian regret for B-MS vs. TS and Fixed Arm base learners with action index $a$ ($T= 10^3, R=10^3, K=M=5$)

Theorems & Definitions (17)

• Definition 3.1: Regret Coefficient
• Lemma 5.1: Good Event
• Lemma 5.2
• Lemma 5.3
• Lemma 5.4
• Theorem 5.5: Oracle-Best Guarantee
• Theorem 6.1
• Corollary A.1
• Lemma A.2
• Lemma A.3