Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Adaptive Experiments: A Unified Approach to Regret Minimization and Best-Arm Identification

Chao Qin, Daniel Russo

TL;DR

This paper introduces a unified framework for adaptive experiments that jointly optimizes within-experiment regret and post-deployment outcomes in large populations. By modeling costs with per-arm within-experiment and post-experiment components and allowing general exponential-family rewards, it derives sharp large-population results that unify classical regret minimization and best-arm identification, culminating in a Lai–Robbins-type cost scaling formula. A key insight is that simple, information-balanced allocation rules, particularly a cost-aware version of Top-two Thompson Sampling, are universally efficient; tuning a single exploitation-rate parameter enables one to trace the Pareto frontier between experiment length and total regret. The Skeptic's Standoff game provides a principled, game-theoretic interpretation of the optimal allocation and the associated equilibrium, linking information-theoretic limits (Chernoff and KL divergences) to deployment decisions. Practically, the results imply substantial reductions in adaptive-experiment duration with limited impact on long-run regret, offering actionable guidance for large-scale online experiments and deployment policies.

Abstract

Practitioners conducting adaptive experiments often encounter two competing priorities: maximizing total welfare (or `reward') through effective treatment assignment and swiftly concluding experiments to implement population-wide treatments. Current literature addresses these priorities separately, with regret minimization studies focusing on the former and best-arm identification research on the latter. This paper bridges this divide by proposing a unified model that simultaneously accounts for within-experiment performance and post-experiment outcomes. We provide a sharp theory of optimal performance in large populations that not only unifies canonical results in the literature but also uncovers novel insights. Our theory reveals that familiar algorithms, such as the recently proposed top-two Thompson sampling algorithm, can optimize a broad class of objectives if a single scalar parameter is appropriately adjusted. In addition, we demonstrate that substantial reductions in experiment duration can often be achieved with minimal impact on both within-experiment and post-experiment regret.

Optimizing Adaptive Experiments: A Unified Approach to Regret Minimization and Best-Arm Identification

TL;DR

This paper introduces a unified framework for adaptive experiments that jointly optimizes within-experiment regret and post-deployment outcomes in large populations. By modeling costs with per-arm within-experiment and post-experiment components and allowing general exponential-family rewards, it derives sharp large-population results that unify classical regret minimization and best-arm identification, culminating in a Lai–Robbins-type cost scaling formula. A key insight is that simple, information-balanced allocation rules, particularly a cost-aware version of Top-two Thompson Sampling, are universally efficient; tuning a single exploitation-rate parameter enables one to trace the Pareto frontier between experiment length and total regret. The Skeptic's Standoff game provides a principled, game-theoretic interpretation of the optimal allocation and the associated equilibrium, linking information-theoretic limits (Chernoff and KL divergences) to deployment decisions. Practically, the results imply substantial reductions in adaptive-experiment duration with limited impact on long-run regret, offering actionable guidance for large-scale online experiments and deployment policies.

Abstract

Practitioners conducting adaptive experiments often encounter two competing priorities: maximizing total welfare (or `reward') through effective treatment assignment and swiftly concluding experiments to implement population-wide treatments. Current literature addresses these priorities separately, with regret minimization studies focusing on the former and best-arm identification research on the latter. This paper bridges this divide by proposing a unified model that simultaneously accounts for within-experiment performance and post-experiment outcomes. We provide a sharp theory of optimal performance in large populations that not only unifies canonical results in the literature but also uncovers novel insights. Our theory reveals that familiar algorithms, such as the recently proposed top-two Thompson sampling algorithm, can optimize a broad class of objectives if a single scalar parameter is appropriately adjusted. In addition, we demonstrate that substantial reductions in experiment duration can often be achieved with minimal impact on both within-experiment and post-experiment regret.
Paper Structure (78 sections, 68 theorems, 269 equations, 3 figures, 5 algorithms)

This paper contains 78 sections, 68 theorems, 269 equations, 3 figures, 5 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Limitations of large-population asymptotics.
  4. Numerical teaser
  5. Policies compared.
  6. Findings and observations.
  7. Literature review
  8. Problem formulation
  9. Restrictions on outcome distributions
  10. Length-regret objectives
  11. Generalized objectives
  12. Large population asymptotics
  13. Optimal cost scaling.
  14. Main Results, specialized to the Gaussian setting
  15. An algorithm template
  16. ...and 63 more sections

Key Result

Lemma 1

Consider a policy $\pi^*$ that satisfies It is universally efficient if and only if, every other consistent policy $\pi \in \Pi$ satisfies the following lower bound:

Figures (3)

  • Figure 1: Numerical performance of policies on the six arm instance $\bm{\theta}=(\theta_1,\theta_2,\theta_3,\theta_4,\theta_5,\theta_6)=(0,0.2,0.4,0.6,0.8,1)$ with population size $n=10^8$.
  • Figure 2: The set of length-regret outcomes that are attainable by a consistent policy. The plot considers the 6-arm instance $\bm{\theta}=(\theta_1,\theta_2,\theta_3,\theta_4,\theta_5,\theta_6)=(0,0.2,0.4,0.6,0.8,1)$. All points on the Pareto frontier are attained by adjusting the exploitation rate $\beta$ of top-two Thompson sampling.
  • Figure : Thompson sampling (TS)

Theorems & Definitions (124)

  • Example 1: Best-arm identification with differentiated sampling costs
  • Definition 1: Consistent policy
  • Definition 2: Universally efficient policy
  • Lemma 1
  • Remark 1: A constraint on the probability of incorrect select
  • Theorem 1: Optimality condition of allocation rules
  • Remark 2
  • Remark 3: Connections to existing allocations
  • Remark 4: Connections to optimal computing budget allocations
  • Theorem 2: Lai-Robbins-type formula
  • ...and 114 more