Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Graph Feedback Bandits with Similar Arms

Han Qi, Guo Fei, Li Zhu

TL;DR

This work introduces graph feedback bandits with an epsilon-similarity structure, where edges connect arms whose means differ by less than $\epsilon$. It develops two UCB-based algorithms, Double-UCB (D-UCB) and Conservative-UCB (C-UCB), and proves regret lower bounds that depend on the dominant/independent-dominating set structure of the feedback graph. The authors extend the framework to ballooning environments with increasing arms, providing regret bounds under distributional assumptions and distribution-free conditions, and they corroborate the theory with experiments in stationary and ballooning settings. The results offer practical guidance for online decision problems with smoothly varying arms, such as recommendations and dynamic content platforms, by exploiting similarity-induced side observations to achieve sublinear regret under broad conditions.

Abstract

In this paper, we study the stochastic multi-armed bandit problem with graph feedback. Motivated by the clinical trials and recommendation problem, we assume that two arms are connected if and only if they are similar (i.e., their means are close enough). We establish a regret lower bound for this novel feedback structure and introduce two UCB-based algorithms: D-UCB with problem-independent regret upper bounds and C-UCB with problem-dependent upper bounds. Leveraging the similarity structure, we also consider the scenario where the number of arms increases over time. Practical applications related to this scenario include Q\&A platforms (Reddit, Stack Overflow, Quora) and product reviews in Amazon and Flipkart. Answers (product reviews) continually appear on the website, and the goal is to display the best answers (product reviews) at the top. When the means of arms are independently generated from some distribution, we provide regret upper bounds for both algorithms and discuss the sub-linearity of bounds in relation to the distribution of means. Finally, we conduct experiments to validate the theoretical results.

Graph Feedback Bandits with Similar Arms

TL;DR

This work introduces graph feedback bandits with an epsilon-similarity structure, where edges connect arms whose means differ by less than . It develops two UCB-based algorithms, Double-UCB (D-UCB) and Conservative-UCB (C-UCB), and proves regret lower bounds that depend on the dominant/independent-dominating set structure of the feedback graph. The authors extend the framework to ballooning environments with increasing arms, providing regret bounds under distributional assumptions and distribution-free conditions, and they corroborate the theory with experiments in stationary and ballooning settings. The results offer practical guidance for online decision problems with smoothly varying arms, such as recommendations and dynamic content platforms, by exploiting similarity-induced side observations to achieve sublinear regret under broad conditions.

Abstract

In this paper, we study the stochastic multi-armed bandit problem with graph feedback. Motivated by the clinical trials and recommendation problem, we assume that two arms are connected if and only if they are similar (i.e., their means are close enough). We establish a regret lower bound for this novel feedback structure and introduce two UCB-based algorithms: D-UCB with problem-independent regret upper bounds and C-UCB with problem-dependent upper bounds. Leveraging the similarity structure, we also consider the scenario where the number of arms increases over time. Practical applications related to this scenario include Q\&A platforms (Reddit, Stack Overflow, Quora) and product reviews in Amazon and Flipkart. Answers (product reviews) continually appear on the website, and the goal is to display the best answers (product reviews) at the top. When the means of arms are independently generated from some distribution, we provide regret upper bounds for both algorithms and discuss the sub-linearity of bounds in relation to the distribution of means. Finally, we conduct experiments to validate the theoretical results.
Paper Structure (27 sections, 15 theorems, 95 equations, 3 figures, 4 algorithms)

This paper contains 27 sections, 15 theorems, 95 equations, 3 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation
  4. Graph Feedback with Similar Arms
  5. Ballooning bandit setting
  6. Stationary Environments
  7. Lower Bounds
  8. Double-UCB
  9. Regret Analysis of Double-UCB
  10. Conservative UCB
  11. Regret Analysis of Conservative UCB
  12. UCB-N
  13. Ballooning Environments
  14. Double-UCB for Ballooning Setting
  15. Regret Analysis
  16. ...and 12 more sections

Key Result

Proposition 4.1

Let $G$ denote the feedback graph with similar arms setting, we have $\gamma(G)=i(G) \geq \frac{\alpha(G)}{2}.$

Figures (3)

  • Figure 1: "UCB-N ($\epsilon=0.1$)": Graph feedback with similarity structure. "UCB-N-Standard ($\epsilon=0.1$)": Graph feedback without similarity structure, but the graph used has roughly the same independence number with the former setting. Settings with $T=10^5,K=10^4$. (a) $\epsilon= 0.1,0.2$ for Gaussian rewards. (b) $\epsilon=0.05,0.1$ for Bernoulli rewards.
  • Figure 2: Settings with $T=10^6,K=10^5,\epsilon=0.01$. Bernoulli rewards (a), Gaussian rewards (b).
  • Figure 3: Ballooning Setting. (a) Gaussian arms with $\mathcal{P}$ as $\mathcal{N}(0,1)$ and $\epsilon=0.1,0.2$. (b) Bernoulli arms with $\mathcal{P}$ as $U(0,1)$ and $\epsilon=0.05,0.1$. (c) Bernoulli arms with $\mathcal{P}$ being the half-triangle distribution and $\epsilon=0.05,0.1$.

Theorems & Definitions (18)

  • Proposition 4.1
  • Theorem 4.2
  • proof
  • Theorem 4.3
  • Corollary 4.4
  • Theorem 4.5
  • Theorem 4.6
  • Remark 4.7
  • Theorem 5.2
  • Corollary 5.3
  • ...and 8 more