Selective Reviews of Bandit Problems in AI via a Statistical View
Pengjie Zhou, Haoyu Wei, Huiming Zhang
TL;DR
The paper surveys the probabilistic and statistical foundations of stochastic bandit problems, including MAB, SCAB, and contextual/continuum variants, with a focus on non-asymptotic tools such as concentration inequalities and regret bounds. It contrasts frequentist and Bayesian exploration–exploitation strategies (ETC, UCB, MOSS, TS, MOTS) and connects SCAB to functional data analysis via smooth reward functions and Gaussian processes. A key contribution is synthesizing non-asymptotic regret analyses and minimax lower bounds to illuminate fundamental limits and guide algorithm design, while highlighting practical implications for sequential decision-making under uncertainty. The work emphasizes how concentration-based methods and Bayesian optimization frameworks inform robust, scalable bandit policies across discrete, contextual, and continuum action spaces. The synthesis has practical impact for online experiments, personalized decision-making, and adaptive design stemming from rigorous non-asymptotic guarantees.
Abstract
Reinforcement Learning (RL) is a widely researched area in artificial intelligence that focuses on teaching agents decision-making through interactions with their environment. A key subset includes stochastic multi-armed bandit (MAB) and continuum-armed bandit (SCAB) problems, which model sequential decision-making under uncertainty. This review outlines the foundational models and assumptions of bandit problems, explores non-asymptotic theoretical tools like concentration inequalities and minimax regret bounds, and compares frequentist and Bayesian algorithms for managing exploration-exploitation trade-offs. Additionally, we explore K-armed contextual bandits and SCAB, focusing on their methodologies and regret analyses. We also examine the connections between SCAB problems and functional data analysis. Finally, we highlight recent advances and ongoing challenges in the field.