Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dual-Directed Algorithm Design for Efficient Pure Exploration

Chao Qin, Wei You

TL;DR

The paper addresses the challenge of pure-exploration beyond best-arm identification by developing a unified dual-directed framework that casts sampling as a maximin problem with dual variables. It introduces Information-Directed Selection (IDS) and a PAN (Pitfall-Adapted Nomination) algorithm template, linking top-two ideas and Thompson sampling through a stationarity/KKT viewpoint to achieve asymptotic optimality across multiple problem formulations (fixed-budget, fixed-confidence, and posterior convergence). The key contributions include a universal optimality result for TTTS-IDS in Gaussian BAI, a modular PAN framework extensible to ε-BAI and thresholding bandits, and practical algorithmic templates with extensive numerical validation showing efficiency gains over existing methods. The framework provides a principled, parameter-free approach to pure exploration with broad applicability, and paves the way for extensions to unknown variances, correlated rewards, and large-scale problems, offering significant practical impact for efficient experimental design and sequential decision-making.

Abstract

While experimental design often focuses on selecting the single best alternative from a finite set (e.g., in ranking and selection or best-arm identification), many pure-exploration problems pursue richer goals. Given a specific goal, adaptive experimentation aims to achieve it by strategically allocating sampling effort, with the underlying sample complexity characterized by a maximin optimization problem. By introducing dual variables, we derive necessary and sufficient conditions for an optimal allocation, yielding a unified algorithm design principle that extends the top-two approach beyond best-arm identification. This principle gives rise to Information-Directed Selection, a hyperparameter-free rule that dynamically evaluates and chooses among candidates based on their current informational value. We prove that, when combined with Information-Directed Selection, top-two Thompson sampling attains asymptotic optimality for Gaussian best-arm identification, resolving a notable open question in the pure-exploration literature. Furthermore, our framework produces asymptotically optimal algorithms for pure-exploration thresholding bandits and $\varepsilon$-best-arm identification (i.e., ranking and selection with probability-of-good-selection guarantees), and more generally establishes a recipe for adapting Thompson sampling across a broad class of pure-exploration problems. Extensive numerical experiments highlight the efficiency of our proposed algorithms compared to existing methods.

Dual-Directed Algorithm Design for Efficient Pure Exploration

TL;DR

The paper addresses the challenge of pure-exploration beyond best-arm identification by developing a unified dual-directed framework that casts sampling as a maximin problem with dual variables. It introduces Information-Directed Selection (IDS) and a PAN (Pitfall-Adapted Nomination) algorithm template, linking top-two ideas and Thompson sampling through a stationarity/KKT viewpoint to achieve asymptotic optimality across multiple problem formulations (fixed-budget, fixed-confidence, and posterior convergence). The key contributions include a universal optimality result for TTTS-IDS in Gaussian BAI, a modular PAN framework extensible to ε-BAI and thresholding bandits, and practical algorithmic templates with extensive numerical validation showing efficiency gains over existing methods. The framework provides a principled, parameter-free approach to pure exploration with broad applicability, and paves the way for extensions to unknown variances, correlated rewards, and large-scale problems, offering significant practical impact for efficient experimental design and sequential decision-making.

Abstract

While experimental design often focuses on selecting the single best alternative from a finite set (e.g., in ranking and selection or best-arm identification), many pure-exploration problems pursue richer goals. Given a specific goal, adaptive experimentation aims to achieve it by strategically allocating sampling effort, with the underlying sample complexity characterized by a maximin optimization problem. By introducing dual variables, we derive necessary and sufficient conditions for an optimal allocation, yielding a unified algorithm design principle that extends the top-two approach beyond best-arm identification. This principle gives rise to Information-Directed Selection, a hyperparameter-free rule that dynamically evaluates and chooses among candidates based on their current informational value. We prove that, when combined with Information-Directed Selection, top-two Thompson sampling attains asymptotic optimality for Gaussian best-arm identification, resolving a notable open question in the pure-exploration literature. Furthermore, our framework produces asymptotically optimal algorithms for pure-exploration thresholding bandits and -best-arm identification (i.e., ranking and selection with probability-of-good-selection guarantees), and more generally establishes a recipe for adapting Thompson sampling across a broad class of pure-exploration problems. Extensive numerical experiments highlight the efficiency of our proposed algorithms compared to existing methods.
Paper Structure (104 sections, 41 theorems, 156 equations, 5 figures, 5 tables, 5 algorithms)

This paper contains 104 sections, 41 theorems, 156 equations, 5 figures, 5 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Motivations
  3. A unified approach to general pure-exploration problems.
  4. Generalizing top-two algorithms.
  5. Generalizing Thompson sampling.
  6. A unified approach to different problem formulations.
  7. Contributions and Organization
  8. Organization.
  9. An Illustrative Example: Indifference-Zone-Free Ranking and Selection in Gaussian Models
  10. Problem Formulation
  11. Top-Two Design Principle as Information Balance Condition
  12. Top-Two Thompson Sampling
  13. Top-Two Algorithms and Information Balance Condition
  14. Information-Directed Selection among Top-Two via Stationarity Condition
  15. Optimality of Top-Two Algorithms with Information-Directed Selection
  16. ...and 89 more sections

Key Result

Theorem 1

An allocation rule is asymptotically optimal for $\bm{\theta}$ if its empirical allocation vector $\bm{p}_t$ converges almost surely to the unique component-wise strictly positive probability vector $\bm{p}^* = (p^*_1,\dots,p^*_K)$ satisfying:

Figures (5)

  • Figure 1: $\mathcal{O}(t^{-1/2})$ convergence to the optimal value, with shaded areas representing the first and third quartiles.
  • Figure 2: Estimated PCS are compared for the proposed algorithms and benchmarks. Top row is the slippage configuration with equal variances, bottom row is the equally-spaced configuration with increasing variances.
  • Figure 3: Fix-budget performance estimated over $10^7$ independent replications.
  • Figure 4: Compare the optimal allocation rate with the empirical allocation rate of the algorithms.
  • Figure 5: The probability of false selection for varying values of $k$ with a fixed budget of $1500$, estimated over $30000$ independent replications.

Theorems & Definitions (82)

  • Definition 1: Optimal allocation rules
  • Theorem 1: A sufficient condition for optimality
  • Remark 1: Equivalent form of stationarity condition
  • Remark 2: Selection versus tuning
  • Theorem 2
  • Definition 2: $\delta$-correct
  • Definition 3: Universal efficiency in fixed confidence
  • Example 1: BAI
  • Example 2: Best-$k$ identification
  • Example 3: Pure-exploration thresholding bandits
  • ...and 72 more