Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cost-Aware Optimal Pairwise Pure Exploration

Di Wu, Chengshuai Shi, Ruida Zhou, Cong Shen

TL;DR

This work builds a general framework for pure exploration in multi-armed bandits with arm-dependent costs, introducing CAET (Cost-Aware Pairwise Exploration Task) to identify pairwise arm relationships while minimizing cumulative cost. It derives a minimax-style lower bound T^*(c, μ) and shows that CAET achieves asymptotic optimality (up to a tunable factor θ) by tracking an estimated optimal sampling proportion and balancing zero-cost arms through an adaptive α. The method extends naturally to regret minimization via an explore-then-commit scheme and provides practical insights through theoretical results and numerical experiments. Overall, the paper advances cost-aware pure exploration by unifying tasks under a single framework, addressing zero-cost challenges, and delivering provably efficient algorithms with strong asymptotic guarantees.

Abstract

Pure exploration is one of the fundamental problems in multi-armed bandits (MAB). However, existing works mostly focus on specific pure exploration tasks, without a holistic view of the general pure exploration problem. This work fills this gap by introducing a versatile framework to study pure exploration, with a focus on identifying the pairwise relationships between targeted arm pairs. Moreover, unlike existing works that only optimize the stopping time (i.e., sample complexity), this work considers that arms are associated with potentially different costs and targets at optimizing the cumulative cost that occurred during learning. Under the general framework of pairwise pure exploration with arm-specific costs, a performance lower bound is derived. Then, a novel algorithm, termed CAET (Cost-Aware Pairwise Exploration Task), is proposed. CAET builds on the track-and-stop principle with a novel design to handle the arm-specific costs, which can potentially be zero and thus represent a very challenging case. Theoretical analyses prove that the performance of CAET approaches the lower bound asymptotically. Special cases are further discussed, including an extension to regret minimization, which is another major focus of MAB. The effectiveness and efficiency of CAET are also verified through experimental results under various settings.

Cost-Aware Optimal Pairwise Pure Exploration

TL;DR

This work builds a general framework for pure exploration in multi-armed bandits with arm-dependent costs, introducing CAET (Cost-Aware Pairwise Exploration Task) to identify pairwise arm relationships while minimizing cumulative cost. It derives a minimax-style lower bound T^*(c, μ) and shows that CAET achieves asymptotic optimality (up to a tunable factor θ) by tracking an estimated optimal sampling proportion and balancing zero-cost arms through an adaptive α. The method extends naturally to regret minimization via an explore-then-commit scheme and provides practical insights through theoretical results and numerical experiments. Overall, the paper advances cost-aware pure exploration by unifying tasks under a single framework, addressing zero-cost challenges, and delivering provably efficient algorithms with strong asymptotic guarantees.

Abstract

Pure exploration is one of the fundamental problems in multi-armed bandits (MAB). However, existing works mostly focus on specific pure exploration tasks, without a holistic view of the general pure exploration problem. This work fills this gap by introducing a versatile framework to study pure exploration, with a focus on identifying the pairwise relationships between targeted arm pairs. Moreover, unlike existing works that only optimize the stopping time (i.e., sample complexity), this work considers that arms are associated with potentially different costs and targets at optimizing the cumulative cost that occurred during learning. Under the general framework of pairwise pure exploration with arm-specific costs, a performance lower bound is derived. Then, a novel algorithm, termed CAET (Cost-Aware Pairwise Exploration Task), is proposed. CAET builds on the track-and-stop principle with a novel design to handle the arm-specific costs, which can potentially be zero and thus represent a very challenging case. Theoretical analyses prove that the performance of CAET approaches the lower bound asymptotically. Special cases are further discussed, including an extension to regret minimization, which is another major focus of MAB. The effectiveness and efficiency of CAET are also verified through experimental results under various settings.

Paper Structure

This paper contains 37 sections, 25 theorems, 195 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORKS
  3. PROBLEM FORMULATION
  4. Bandits with Arm-Dependent Costs
  5. The Pairwise Pure Exploration Problem
  6. Cost-Aware Pure Exploration with Fixed Confidence
  7. Preliminaries and Notations
  8. Motivation Applications
  9. LOWER BOUND
  10. CAET ALGORITHM
  11. Sampling Rule
  12. Stopping Rule
  13. Decision Rule
  14. UPPER BOUND
  15. Convergence to the Optimal Proportion
  16. ...and 22 more sections

Key Result

Theorem 1

Let $\delta \in (0,1)$ and give a pairwise exploration task $(\mathcal{G},\varphi)$. For any $\delta$-PAC strategy and any bandit model $\boldsymbol{\mu}\in S$, the following holds: where

Figures (1)

  • Figure 1: Experimental results

Theorems & Definitions (31)

  • Definition 1: Pairwise Exploration Task
  • Example 1: Ranking Identification
  • Example 2: BAI
  • Example 3: Best-$m$-arms Identification
  • Theorem 1
  • Proposition 1: Informational Threshold
  • Proposition 2: Deviational Threshold
  • Theorem 2: Upper Bound
  • Lemma 1
  • Lemma 2: Lemma 1 in kaufmann2016complexity
  • ...and 21 more