Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DART: aDaptive Accept RejecT for non-linear top-K subset identification

Mridul Agarwal, Vaneet Aggarwal, Christopher J. Quinn, Abhishek Umrawal

TL;DR

A novel algorithm for the combinatorial setting without using individual arm feedback or requiring linearity of the reward function is presented, which significantly outperforms existing methods for both linear and non-linear joint reward environments.

Abstract

We consider the bandit problem of selecting $K$ out of $N$ arms at each time step. The reward can be a non-linear function of the rewards of the selected individual arms. The direct use of a multi-armed bandit algorithm requires choosing among $\binom{N}{K}$ options, making the action space large. To simplify the problem, existing works on combinatorial bandits {typically} assume feedback as a linear function of individual rewards. In this paper, we prove the lower bound for top-$K$ subset selection with bandit feedback with possibly correlated rewards. We present a novel algorithm for the combinatorial setting without using individual arm feedback or requiring linearity of the reward function. Additionally, our algorithm works on correlated rewards of individual arms. Our algorithm, aDaptive Accept RejecT (DART), sequentially finds good arms and eliminates bad arms based on confidence bounds. DART is computationally efficient and uses storage linear in $N$. Further, DART achieves a regret bound of $\tilde{\mathcal{O}}(K\sqrt{KNT})$ for a time horizon $T$, which matches the lower bound in bandit feedback up to a factor of $\sqrt{\log{2NT}}$. When applied to the problem of cross-selling optimization and maximizing the mean of individual rewards, the performance of the proposed algorithm surpasses that of state-of-the-art algorithms. We also show that DART significantly outperforms existing methods for both linear and non-linear joint reward environments.

DART: aDaptive Accept RejecT for non-linear top-K subset identification

TL;DR

A novel algorithm for the combinatorial setting without using individual arm feedback or requiring linearity of the reward function is presented, which significantly outperforms existing methods for both linear and non-linear joint reward environments.

Abstract

We consider the bandit problem of selecting out of arms at each time step. The reward can be a non-linear function of the rewards of the selected individual arms. The direct use of a multi-armed bandit algorithm requires choosing among options, making the action space large. To simplify the problem, existing works on combinatorial bandits {typically} assume feedback as a linear function of individual rewards. In this paper, we prove the lower bound for top- subset selection with bandit feedback with possibly correlated rewards. We present a novel algorithm for the combinatorial setting without using individual arm feedback or requiring linearity of the reward function. Additionally, our algorithm works on correlated rewards of individual arms. Our algorithm, aDaptive Accept RejecT (DART), sequentially finds good arms and eliminates bad arms based on confidence bounds. DART is computationally efficient and uses storage linear in . Further, DART achieves a regret bound of for a time horizon , which matches the lower bound in bandit feedback up to a factor of . When applied to the problem of cross-selling optimization and maximizing the mean of individual rewards, the performance of the proposed algorithm surpasses that of state-of-the-art algorithms. We also show that DART significantly outperforms existing methods for both linear and non-linear joint reward environments.

Paper Structure

This paper contains 23 sections, 9 theorems, 46 equations, 3 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation
  4. Lower bound on top-$K$ subset identification
  5. Proposed DART Algorithm
  6. Regret Analysis
  7. Number of samples to move an arm in $\mathcal{N}_e$ to either of $\mathcal{A}_e$ or $\mathcal{R}_e$
  8. Regret from sampling sub-optimal arms
  9. Experiments
  10. Reward Setups
  11. Baseline Algorithms
  12. Experimental Details
  13. Results and Discussion
  14. Conclusion
  15. Lower bound on the top-$K$ subset selection problem
  16. ...and 8 more sections

Key Result

Corollary 1

For any arms $i, j\in\mathcal{N}$ and any subset $\bm{S}\subset \mathcal{N}\setminus \{i,j\}$ of size $K-1$,

Figures (3)

  • Figure 1: Regret plots for joint rewards as the mean of individual arm rewards.
  • Figure 2: Regret plots for joint rewards as a quadratic function of individual arm rewards.
  • Figure 3: Regret plots for joint rewards as max of individual arm rewards

Theorems & Definitions (10)

  • Corollary 1
  • Theorem 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Lemma 6
  • Theorem 7
  • Remark 8
  • Lemma 9: Azuma-Hoeffding bercu2015concentration
  • Lemma 10