Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Introduction to Multi-Armed Bandits

Aleksandrs Slivkins

TL;DR

This work surveys the multi-armed bandit framework across stochastic, Bayesian, Lipschitz, and adversarial settings. It presents practical algorithms (Explore-first, Epsilon-greedy, UCB, Successive Elimination, Thompson Sampling, Zooming) and analyzes their regret, including instance-dependent and horizon-free bounds. The text develops fundamental lower bounds via KL-divergence, introduces bandits with similarity information, and extends to full feedback and adversarial models with Hedge and Exp4. It highlights the role of priors, confidence bounds, discretization, and adaptive strategies to manage exploration-exploitation in diverse environments. Together, these chapters map a comprehensive landscape of bandit algorithms, their theoretical limits, and pertinent extensions to economics, pricing, and online learning with experts.

Abstract

Multi-armed bandits a simple but very powerful framework for algorithms that make decisions over time under uncertainty. An enormous body of work has accumulated over the years, covered in several books and surveys. This book provides a more introductory, textbook-like treatment of the subject. Each chapter tackles a particular line of work, providing a self-contained, teachable technical introduction and a brief review of the further developments; many of the chapters conclude with exercises. The book is structured as follows. The first four chapters are on IID rewards, from the basic model to impossibility results to Bayesian priors to Lipschitz rewards. The next three chapters cover adversarial rewards, from the full-feedback version to adversarial bandits to extensions with linear rewards and combinatorially structured actions. Chapter 8 is on contextual bandits, a middle ground between IID and adversarial bandits in which the change in reward distributions is completely explained by observable contexts. The last three chapters cover connections to economics, from learning in repeated games to bandits with supply/budget constraints to exploration in the presence of incentives. The appendix provides sufficient background on concentration and KL-divergence. The chapters on "bandits with similarity information", "bandits with knapsacks" and "bandits and agents" can also be consumed as standalone surveys on the respective topics.

Introduction to Multi-Armed Bandits

TL;DR

This work surveys the multi-armed bandit framework across stochastic, Bayesian, Lipschitz, and adversarial settings. It presents practical algorithms (Explore-first, Epsilon-greedy, UCB, Successive Elimination, Thompson Sampling, Zooming) and analyzes their regret, including instance-dependent and horizon-free bounds. The text develops fundamental lower bounds via KL-divergence, introduces bandits with similarity information, and extends to full feedback and adversarial models with Hedge and Exp4. It highlights the role of priors, confidence bounds, discretization, and adaptive strategies to manage exploration-exploitation in diverse environments. Together, these chapters map a comprehensive landscape of bandit algorithms, their theoretical limits, and pertinent extensions to economics, pricing, and online learning with experts.

Abstract

Multi-armed bandits a simple but very powerful framework for algorithms that make decisions over time under uncertainty. An enormous body of work has accumulated over the years, covered in several books and surveys. This book provides a more introductory, textbook-like treatment of the subject. Each chapter tackles a particular line of work, providing a self-contained, teachable technical introduction and a brief review of the further developments; many of the chapters conclude with exercises. The book is structured as follows. The first four chapters are on IID rewards, from the basic model to impossibility results to Bayesian priors to Lipschitz rewards. The next three chapters cover adversarial rewards, from the full-feedback version to adversarial bandits to extensions with linear rewards and combinatorially structured actions. Chapter 8 is on contextual bandits, a middle ground between IID and adversarial bandits in which the change in reward distributions is completely explained by observable contexts. The last three chapters cover connections to economics, from learning in repeated games to bandits with supply/budget constraints to exploration in the presence of incentives. The appendix provides sufficient background on concentration and KL-divergence. The chapters on "bandits with similarity information", "bandits with knapsacks" and "bandits and agents" can also be consumed as standalone surveys on the respective topics.

Paper Structure

This paper contains 120 sections, 89 theorems, 384 equations, 8 figures, 3 tables, 23 algorithms.

Table of Contents

  1. Stochastic Bandits
  2. Model and examples
  3. Simple algorithms: uniform exploration
  4. Improvement: Epsilon-greedy algorithm
  5. Non-adaptive exploration
  6. Advanced algorithms: adaptive exploration
  7. Clean event and confidence bounds
  8. Successive Elimination algorithm
  9. Optimism under uncertainty
  10. Forward look: bandits with initial information
  11. Literature review and discussion
  12. Exercises and hints
  13. Lower Bounds
  14. Background on KL-divergence
  15. A simple example: flipping one coin
  16. ...and 105 more sections

Key Result

Theorem 1.5

Explore-first achieves regret $\operatornamewithlimits{\mathbb{E}}\left[\,R(T)\,\right] \leq T^{2/3} \times O(K \log T)^{1/3}$.

Figures (8)

  • Figure 1: the $j$-th cell contains the reward of the $j$-th time we pull arm $a$, i.e., reward of arm $a$ when $n_t(a)=j$
  • Figure 2: $t$ is the last round that the two confidence intervals still overlap
  • Figure 3: Discretization of the context space
  • Figure 4: The learning loop.
  • Figure 5: Example: alternating arm is twice as good as a fixed arm
  • ...and 3 more figures

Theorems & Definitions (210)

  • Remark 1.1
  • Remark 1.2: Terminology
  • Remark 1.3
  • Remark 1.4
  • Theorem 1.5
  • Theorem 1.6
  • Definition 1.7
  • Lemma 1.8
  • Remark 1.9
  • Theorem 1.10
  • ...and 200 more