Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Risk and optimal policies in bandit experiments

Karun Adusumilli

TL;DR

The paper develops a diffusion-asymptotics framework for bandit experiments, proving that the minimal Bayes risk under $n^{-1/2}$-scaled rewards is characterized by a $2^{\textrm{nd}}$-order PDE and that this characterization extends from Gaussian to parametric and non-parametric reward models via a limit-of-experiments argument. It shows that, asymptotically, only two state variables per arm are needed, enabling practical computation of Bayes and minimax policies that systematically outperform Thompson sampling, with tuned MOSS offering near-minimax performance in several settings. The work provides existence/uniqueness results for the PDE, convergence of discrete schemes to the PDE solution, and constructive ways to implement near-optimal policies through piecewise-constant controls and Monte-Carlo or finite-difference computation. Empirically, the Bayes-optimal policies dominate standard bandit algorithms in both one- and two-armed illustrated examples, underscoring the practical impact of the PDE-based approach for adaptive experimentation and decision making. The framework also accommodates extensions such as discounting and best-arm identification, broadening its relevance to diverse sequential experimentation problems.

Abstract

We provide a decision theoretic analysis of bandit experiments under local asymptotics. Working within the framework of diffusion processes, we define suitable notions of asymptotic Bayes and minimax risk for these experiments. For normally distributed rewards, the minimal Bayes risk can be characterized as the solution to a second-order partial differential equation (PDE). Using a limit of experiments approach, we show that this PDE characterization also holds asymptotically under both parametric and non-parametric distributions of the rewards. The approach further describes the state variables it is asymptotically sufficient to restrict attention to, and thereby suggests a practical strategy for dimension reduction. The PDEs characterizing minimal Bayes risk can be solved efficiently using sparse matrix routines or Monte-Carlo methods. We derive the optimal Bayes and minimax policies from their numerical solutions. These optimal policies substantially dominate existing methods such as Thompson sampling; the risk of the latter is often twice as high.

Risk and optimal policies in bandit experiments

TL;DR

The paper develops a diffusion-asymptotics framework for bandit experiments, proving that the minimal Bayes risk under -scaled rewards is characterized by a -order PDE and that this characterization extends from Gaussian to parametric and non-parametric reward models via a limit-of-experiments argument. It shows that, asymptotically, only two state variables per arm are needed, enabling practical computation of Bayes and minimax policies that systematically outperform Thompson sampling, with tuned MOSS offering near-minimax performance in several settings. The work provides existence/uniqueness results for the PDE, convergence of discrete schemes to the PDE solution, and constructive ways to implement near-optimal policies through piecewise-constant controls and Monte-Carlo or finite-difference computation. Empirically, the Bayes-optimal policies dominate standard bandit algorithms in both one- and two-armed illustrated examples, underscoring the practical impact of the PDE-based approach for adaptive experimentation and decision making. The framework also accommodates extensions such as discounting and best-arm identification, broadening its relevance to diverse sequential experimentation problems.

Abstract

We provide a decision theoretic analysis of bandit experiments under local asymptotics. Working within the framework of diffusion processes, we define suitable notions of asymptotic Bayes and minimax risk for these experiments. For normally distributed rewards, the minimal Bayes risk can be characterized as the solution to a second-order partial differential equation (PDE). Using a limit of experiments approach, we show that this PDE characterization also holds asymptotically under both parametric and non-parametric distributions of the rewards. The approach further describes the state variables it is asymptotically sufficient to restrict attention to, and thereby suggests a practical strategy for dimension reduction. The PDEs characterizing minimal Bayes risk can be solved efficiently using sparse matrix routines or Monte-Carlo methods. We derive the optimal Bayes and minimax policies from their numerical solutions. These optimal policies substantially dominate existing methods such as Thompson sampling; the risk of the latter is often twice as high.
Paper Structure (45 sections, 17 theorems, 160 equations, 5 figures, 2 algorithms)

This paper contains 45 sections, 17 theorems, 160 equations, 5 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Diffusion asymptotics and statistical risk
  3. Payoff and loss functions
  4. Bayes risk
  5. Priors and posteriors
  6. PDE characterization of Bayes and minimal Bayes risk
  7. A special case: one-armed bandits
  8. Comparison with existing methods
  9. Minimax risk
  10. Formal properties under gaussian rewards
  11. Existence and uniqueness of PDE solutions
  12. Convergence to the PDE solution
  13. Piece-wise constant policies and batched bandits
  14. Algorithms and empirical illustrations
  15. Algorithms
  16. ...and 30 more sections

Key Result

Theorem 1

barles2007error Suppose $\mu^{+}(\cdot),\mu(\cdot)$ are $\gamma$-Hölder continuous for some $\gamma>0$. Then there exists a unique, $\gamma$-Hölder continuous viscosity solution to PDE (eq:PDE optimal bayes risk).

Figures (5)

  • Figure 2.1: Asymptotic risk of various policies under one-armed bandits
  • Figure 2.2: Policy maps for a one-armed bandit
  • Figure 4.1: Empirical illustration - one-armed bandit
  • Figure 4.2: Empirical illustration - two-armed bandits
  • Figure B.1: Monte-Carlo simulations

Theorems & Definitions (26)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • ...and 16 more