Bayesian Design Principles for Frequentist Sequential Learning
Yunbei Xu, Assaf Zeevi
TL;DR
This work proposes Algorithmic Information Ratio (AIR) as a principled, prior-free Bayesian-style framework for sequential learning with partial feedback, unifying frequentist regret analysis with constructive Bayesian-belief design. By maximizing AIR, the authors generate round-by-round algorithmic beliefs that balance exploration and exploitation and lead to regret guarantees matching the best-known priors-based results across stochastic, adversarial, and non-stationary environments. They introduce Model-index AIR (MAIR) for stochastic settings and derive constructive algorithms—Adaptive Posterior Sampling (APS) and Adaptive Minimax Sampling (AMS)—with closed-form implementations in key cases like Bernoulli MAB and Gaussian linear bandits. The framework extends to linear bandits, bandit convex optimization, and reinforcement learning, delivering near-optimal rates (e.g., $O( ilde{d}^{2.5}\sqrt{T})$ for bandit convex optimization) with finite-time, poly-time algorithms. Empirically, a Bernoulli MAB instance demonstrates “best-of-all-worlds” performance, outperforming UCB and EXP3 across regimes, illustrating the practical impact of AIR-based principled design for real-world sequential decision problems.
Abstract
We develop a general theory to optimize the frequentist regret for sequential learning problems, where efficient bandit and reinforcement learning algorithms can be derived from unified Bayesian principles. We propose a novel optimization approach to generate "algorithmic beliefs" at each round, and use Bayesian posteriors to make decisions. The optimization objective to create "algorithmic beliefs," which we term "Algorithmic Information Ratio," represents an intrinsic complexity measure that effectively characterizes the frequentist regret of any algorithm. To the best of our knowledge, this is the first systematical approach to make Bayesian-type algorithms prior-free and applicable to adversarial settings, in a generic and optimal manner. Moreover, the algorithms are simple and often efficient to implement. As a major application, we present a novel algorithm for multi-armed bandits that achieves the "best-of-all-worlds" empirical performance in the stochastic, adversarial, and non-stationary environments. And we illustrate how these principles can be used in linear bandits, bandit convex optimization, and reinforcement learning.