Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Multi-Objective Best Arm Identification with Fixed Confidence

Zhirui Chen, P. N. Karthik, Yeow Meng Chee, Vincent Y. F. Tan

TL;DR

The paper addresses multi-objective best arm identification under fixed confidence, where each pull yields an $M$-dimensional reward and the goal is to identify the best arm for every objective. It introduces a problem-dependent lower bound $c^*(v)$ and a novel surrogate-proportion MO-BAI algorithm that avoids per-step max–min optimization yet achieves asymptotic optimality up to a factor $(1+ ext{e})$, confirmed both theoretically and empirically. The method leverages surrogate linearizations and a Chernoff-type stopping rule to guarantee $ ext{PAC}$ correctness while minimizing sample complexity, outperforming baselines on synthetic and real datasets. These results extend pure exploration in MO-MAB beyond Pareto-frontier identification and offer a practical, scalable approach for vector-valued rewards in applications like targeted advertising and multi-criteria decision making.

Abstract

We consider a multi-armed bandit setting with finitely many arms, in which each arm yields an $M$-dimensional vector reward upon selection. We assume that the reward of each dimension (a.k.a. {\em objective}) is generated independently of the others. The best arm of any given objective is the arm with the largest component of mean corresponding to the objective. The end goal is to identify the best arm of {\em every} objective in the shortest (expected) time subject to an upper bound on the probability of error (i.e., fixed-confidence regime). We establish a problem-dependent lower bound on the limiting growth rate of the expected stopping time, in the limit of vanishing error probabilities. This lower bound, we show, is characterised by a max-min optimisation problem that is computationally expensive to solve at each time step. We propose an algorithm that uses the novel idea of {\em surrogate proportions} to sample the arms at each time step, eliminating the need to solve the max-min optimisation problem at each step. We demonstrate theoretically that our algorithm is asymptotically optimal. In addition, we provide extensive empirical studies to substantiate the efficiency of our algorithm. While existing works on pure exploration with multi-objective multi-armed bandits predominantly focus on {\em Pareto frontier identification}, our work fills the gap in the literature by conducting a formal investigation of the multi-objective best arm identification problem.

Optimal Multi-Objective Best Arm Identification with Fixed Confidence

TL;DR

The paper addresses multi-objective best arm identification under fixed confidence, where each pull yields an -dimensional reward and the goal is to identify the best arm for every objective. It introduces a problem-dependent lower bound and a novel surrogate-proportion MO-BAI algorithm that avoids per-step max–min optimization yet achieves asymptotic optimality up to a factor , confirmed both theoretically and empirically. The method leverages surrogate linearizations and a Chernoff-type stopping rule to guarantee correctness while minimizing sample complexity, outperforming baselines on synthetic and real datasets. These results extend pure exploration in MO-MAB beyond Pareto-frontier identification and offer a practical, scalable approach for vector-valued rewards in applications like targeted advertising and multi-criteria decision making.

Abstract

We consider a multi-armed bandit setting with finitely many arms, in which each arm yields an -dimensional vector reward upon selection. We assume that the reward of each dimension (a.k.a. {\em objective}) is generated independently of the others. The best arm of any given objective is the arm with the largest component of mean corresponding to the objective. The end goal is to identify the best arm of {\em every} objective in the shortest (expected) time subject to an upper bound on the probability of error (i.e., fixed-confidence regime). We establish a problem-dependent lower bound on the limiting growth rate of the expected stopping time, in the limit of vanishing error probabilities. This lower bound, we show, is characterised by a max-min optimisation problem that is computationally expensive to solve at each time step. We propose an algorithm that uses the novel idea of {\em surrogate proportions} to sample the arms at each time step, eliminating the need to solve the max-min optimisation problem at each step. We demonstrate theoretically that our algorithm is asymptotically optimal. In addition, we provide extensive empirical studies to substantiate the efficiency of our algorithm. While existing works on pure exploration with multi-objective multi-armed bandits predominantly focus on {\em Pareto frontier identification}, our work fills the gap in the literature by conducting a formal investigation of the multi-objective best arm identification problem.
Paper Structure (29 sections, 21 theorems, 144 equations, 2 figures, 3 tables, 4 algorithms)

This paper contains 29 sections, 21 theorems, 144 equations, 2 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Overview of Existing Works
  3. Our Contributions
  4. Preliminaries and Problem Setup
  5. Lower bound
  6. Achievability: Proposed Method
  7. Surrogate Proportion and Arm Selection Rule
  8. Stopping and Recommendation Rules
  9. Performance of MO-BAI
  10. Proof Sketch of Theorem \ref{['thm:upperbound']}
  11. Bounding the Limiting Value of Z(t)/t
  12. Bounding the Stopping Time taudelta
  13. Numerical Study
  14. Conclusions and Future Work
  15. More Details on Numerical Study
  16. ...and 14 more sections

Key Result

Proposition 3.1

Fix $\delta \in (0,1)$. For any $\delta$-PAC policy $\pi$, where the constant $c^*(v)$ is given by In eq:cstar, $\Gamma$ denotes the set of probability distributions on $[K]$, and we use the convention $\frac{\omega_i \, \omega_{i_m^*(v)}}{\omega_i+\omega_{i_m^*(v)}} = 0$ if $\omega_i = \omega_{i_m^*(v)} = 0$. Consequently, taking limits as $\delta \downarrow 0$ in eq:lower-bound, we get

Figures (2)

  • Figure 1: Plot of average stopping times of MO-BAI and Baseline (with varying iteration numbers) for the synthetic dataset.
  • Figure 2: Comparison of the empirical stopping times with varying values of $\eta$ for the synthetic dataset.

Theorems & Definitions (46)

  • Proposition 3.1
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • Remark 6
  • Proposition 4.1
  • Theorem 4.2
  • Remark 7
  • ...and 36 more