Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A General Framework for Clustering and Distribution Matching with Bandit Feedback

Recep Can Yavas, Yuqi Huang, Vincent Y. F. Tan, Jonathan Scarlett

TL;DR

The paper develops a general online framework for clustering and distribution matching with bandit feedback on a finite alphabet. It unifies problems like matching pairs, generalized odd-arm identification, and N-ary clustering under a fixed-confidence setting, and proves a non-asymptotic lower bound while proposing TaS-FW, a Track-and-Stop algorithm based on Frank--Wolfe that achieves asymptotic optimality with a novel second-order convergence term. The approach combines C-tracking, an initial uniform sampling phase, and a Frank--Wolfe-driven oracle to adaptively allocate pulls and decide stopping via a GLLR-inspired statistic. Theoretical results include a leading-term lower bound on the stopping time and a matching upper bound for TaS-FW, plus precise concentration and curvature analyses. Empirical results across multiple examples and a Gaussian extension corroborate the theoretical findings, highlighting competitive sample efficiency and the trade-off between computation time and performance. The work advances principled, scalable online clustering under bandit feedback with finite-alphabet arms, with implications for rapid partition discovery in adaptive experimentation and targeted decision-making.

Abstract

We develop a general framework for clustering and distribution matching problems with bandit feedback. We consider a $K$-armed bandit model where some subset of $K$ arms is partitioned into $M$ groups. Within each group, the random variable associated to each arm follows the same distribution on a finite alphabet. At each time step, the decision maker pulls an arm and observes its outcome from the random variable associated to that arm. Subsequent arm pulls depend on the history of arm pulls and their outcomes. The decision maker has no knowledge of the distributions of the arms or the underlying partitions. The task is to devise an online algorithm to learn the underlying partition of arms with the least number of arm pulls on average and with an error probability not exceeding a pre-determined value~$δ$. Several existing problems fall under our general framework, including finding $M$ pairs of arms, odd arm identification, and $N$-ary clustering of $K$ arms belong to our general framework. We derive a non-asymptotic lower bound on the average number of arm pulls for any online algorithm with an error probability not exceeding $δ$. Furthermore, we develop a computationally-efficient online algorithm based on the Track-and-Stop method and Frank--Wolfe algorithm, and show that the average number of arm pulls of our algorithm asymptotically matches that of the lower bound. Our refined analysis also uncovers a novel bound on the speed at which the average number of arm pulls of our algorithm converges to the fundamental limit as $δ$ vanishes.

A General Framework for Clustering and Distribution Matching with Bandit Feedback

TL;DR

The paper develops a general online framework for clustering and distribution matching with bandit feedback on a finite alphabet. It unifies problems like matching pairs, generalized odd-arm identification, and N-ary clustering under a fixed-confidence setting, and proves a non-asymptotic lower bound while proposing TaS-FW, a Track-and-Stop algorithm based on Frank--Wolfe that achieves asymptotic optimality with a novel second-order convergence term. The approach combines C-tracking, an initial uniform sampling phase, and a Frank--Wolfe-driven oracle to adaptively allocate pulls and decide stopping via a GLLR-inspired statistic. Theoretical results include a leading-term lower bound on the stopping time and a matching upper bound for TaS-FW, plus precise concentration and curvature analyses. Empirical results across multiple examples and a Gaussian extension corroborate the theoretical findings, highlighting competitive sample efficiency and the trade-off between computation time and performance. The work advances principled, scalable online clustering under bandit feedback with finite-alphabet arms, with implications for rapid partition discovery in adaptive experimentation and targeted decision-making.

Abstract

We develop a general framework for clustering and distribution matching problems with bandit feedback. We consider a -armed bandit model where some subset of arms is partitioned into groups. Within each group, the random variable associated to each arm follows the same distribution on a finite alphabet. At each time step, the decision maker pulls an arm and observes its outcome from the random variable associated to that arm. Subsequent arm pulls depend on the history of arm pulls and their outcomes. The decision maker has no knowledge of the distributions of the arms or the underlying partitions. The task is to devise an online algorithm to learn the underlying partition of arms with the least number of arm pulls on average and with an error probability not exceeding a pre-determined value~. Several existing problems fall under our general framework, including finding pairs of arms, odd arm identification, and -ary clustering of arms belong to our general framework. We derive a non-asymptotic lower bound on the average number of arm pulls for any online algorithm with an error probability not exceeding . Furthermore, we develop a computationally-efficient online algorithm based on the Track-and-Stop method and Frank--Wolfe algorithm, and show that the average number of arm pulls of our algorithm asymptotically matches that of the lower bound. Our refined analysis also uncovers a novel bound on the speed at which the average number of arm pulls of our algorithm converges to the fundamental limit as vanishes.
Paper Structure (50 sections, 19 theorems, 177 equations, 6 figures, 1 table, 1 algorithm)

This paper contains 50 sections, 19 theorems, 177 equations, 6 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. The Lower Bound and Track-and-Stop Algorithms
  4. Contributions
  5. Problem Formulation
  6. Notation
  7. Problem Statement
  8. Example 1 (Matching pairs from two groups of arms)
  9. Example 2 (Generalization of the odd arm identification problem)
  10. Example 3 (N-ary clustering of K arms)
  11. A Converse (Lower) Bound
  12. A Track-and-Stop Strategy Based on Frank--Wolfe Algorithm (TaS-FW) and an Achievability (Upper) Bound
  13. C-tracking
  14. Uniform sampling phase
  15. FWS phase with forced exploration
  16. ...and 35 more sections

Key Result

Lemma 1

Consider $X^N$ and the hypotheses $H_0$ and $H_1$ as defined in eq:hypoH0--eq:hypoH1. Let $w_i = \frac{n_i}{N}$ for $i \in [B]$. Denote $\hat{P}_{[B]} = (\hat{P}_{X_i^{n_i}})_{i \in [B]}$. Then,

Figures (6)

  • Figure 1: The examples of matching pairs, odd arm identification, and $N$-ary clustering of $K$ arms are illustrated. Each shape indicates a unique distribution. Arms that are demonstrated by the same shape are in the same cluster. The number of arms for each example is $K = 8$. For Examples 1, the number of clusters is $M = 3$. In Example 1, the decision maker knows that arms 1, 2, and 3 are the nominal arms that must appear in $M = 3$ clusters. In Example 2, the decision maker knows that exactly one arm has a different distribution than the others. In Example 3, the decision maker knows that there are $N = 3$ individual groups but does not know about the size or the content of each group.
  • Figure 2: Example 1: Matching pairs with two groups of arms with $|\mathcal{X}| = 3$.
  • Figure 3: Example 1: Matching pairs with two groups of arms with $|\mathcal{X}| = 5$.
  • Figure 4: Example 2: Odd arm identification.
  • Figure 5: Example 3: $N$-ary clustering of $K$ arms.
  • ...and 1 more figures

Theorems & Definitions (23)

  • Definition 1
  • Lemma 1
  • Lemma 2
  • Remark 1
  • Remark 2
  • Theorem 1
  • Theorem 2
  • Lemma 3: garivier2016
  • Corollary 1
  • Lemma 4
  • ...and 13 more