Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preference-based Pure Exploration

Apurv Shukla, Debabrota Basu

TL;DR

This work studies Prefence-based Pure Exploration (PrePEx) for bandits with vector-valued rewards ordered by a convex cone $\mathcal{C}$, aiming to exactly identify the Pareto Optimal Set $\mathcal{P}^*(M)$ with fixed confidence. It derives a KL-divergence based lower bound on the stopping time that sensitively depends on the geometry of $\mathcal{C}$ and the policy gaps, and specializes this bound to Gaussian rewards to reveal a bilinear projection structure. The authors then design PreTS, a Track-and-Stop algorithm that uses a convex relaxation of the lower bound to compute sampling allocations and a Chernoff-type stopping rule, together with a new vector-valued concentration result to certify correctness. They prove that PreTS achieves the convexified lower bound asymptotically, i.e., its sample complexity matches the lower bound up to constants as $\delta \to 0$. The methodology enables efficient, exact Pareto-front identification under user-defined preferences and provides foundational insights for downstream multi-objective decision-making under uncertainty.

Abstract

We study the preference-based pure exploration problem for bandits with vector-valued rewards. The rewards are ordered using a (given) preference cone $\mathcal{C}$ and our goal is to identify the set of Pareto optimal arms. First, to quantify the impact of preferences, we derive a novel lower bound on sample complexity for identifying the most preferred policy with a confidence level $1-δ$. Our lower bound elicits the role played by the geometry of the preference cone and punctuates the difference in hardness compared to existing best-arm identification variants of the problem. We further explicate this geometry when the rewards follow Gaussian distributions. We then provide a convex relaxation of the lower bound and leverage it to design the Preference-based Track and Stop (PreTS) algorithm that identifies the most preferred policy. Finally, we show that the sample complexity of PreTS is asymptotically tight by deriving a new concentration inequality for vector-valued rewards.

Preference-based Pure Exploration

TL;DR

This work studies Prefence-based Pure Exploration (PrePEx) for bandits with vector-valued rewards ordered by a convex cone , aiming to exactly identify the Pareto Optimal Set with fixed confidence. It derives a KL-divergence based lower bound on the stopping time that sensitively depends on the geometry of and the policy gaps, and specializes this bound to Gaussian rewards to reveal a bilinear projection structure. The authors then design PreTS, a Track-and-Stop algorithm that uses a convex relaxation of the lower bound to compute sampling allocations and a Chernoff-type stopping rule, together with a new vector-valued concentration result to certify correctness. They prove that PreTS achieves the convexified lower bound asymptotically, i.e., its sample complexity matches the lower bound up to constants as . The methodology enables efficient, exact Pareto-front identification under user-defined preferences and provides foundational insights for downstream multi-objective decision-making under uncertainty.

Abstract

We study the preference-based pure exploration problem for bandits with vector-valued rewards. The rewards are ordered using a (given) preference cone and our goal is to identify the set of Pareto optimal arms. First, to quantify the impact of preferences, we derive a novel lower bound on sample complexity for identifying the most preferred policy with a confidence level . Our lower bound elicits the role played by the geometry of the preference cone and punctuates the difference in hardness compared to existing best-arm identification variants of the problem. We further explicate this geometry when the rewards follow Gaussian distributions. We then provide a convex relaxation of the lower bound and leverage it to design the Preference-based Track and Stop (PreTS) algorithm that identifies the most preferred policy. Finally, we show that the sample complexity of PreTS is asymptotically tight by deriving a new concentration inequality for vector-valued rewards.

Paper Structure

This paper contains 26 sections, 19 theorems, 79 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preference-based Pure Exploration Problem
  4. Lower Bound on Sample Complexity
  5. Characteraization of Lower Bounds for Gaussians
  6. Algorithm Design: PreTS
  7. Convex Relaxation of the Lower Bound
  8. Algorithm: Preference-based Track-and-Stop (PreTS)
  9. Upper Bound on Sample Complexity
  10. Conclusion and Future Works
  11. Appendix
  12. Notations
  13. Proofs of Lower Bounds
  14. Generic Lower Bound: Proof of Theorem \ref{['thm:lower-bound']}
  15. Lower Bound for Gaussians: Proof of Theorem \ref{['thm:gaussian-bandit']}
  16. ...and 11 more sections

Key Result

Theorem 3.1

Given a bandit model $M \in \mathcal{M}$, a preference cone $\mathcal{C}$, and a confidence level $\delta \in [0,1)$, the expected stopping time of any $(1-\delta)$-correct PrePEx algorithm, to identify the Pareto Optimal Set is where, the expectation is taken over the stochasticity of both the algorithm and the bandit instance. Here, $\mathcal{T}_{M,\mathcal{C}}$ is called the characteristic tim

Figures (1)

  • Figure 1: Effect of cone selection on size of Pareto optimal set

Theorems & Definitions (37)

  • Definition 2.1: Ordering Cone
  • Definition 2.2: Polyhederal Ordering Cone
  • Example 2.1: Preference cones
  • Definition 2.3: Partial Order
  • Definition 2.4: Order over arms
  • Definition 2.5: Pareto Optimal Set
  • Example 2.2: Pareto Optimal Sets for different cones
  • Definition 2.6: $(1-\delta)$-correct PrePEX
  • Theorem 3.1: Lower Bound
  • Theorem 3.2: Lower Bound for Gaussian Bandits
  • ...and 27 more