Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HELLINGER-UCB: A novel algorithm for stochastic multi-armed bandit problem and cold start problem in recommender system

Ruibo Yang, Jiazhou Wang, Andrew Mullhaupt

TL;DR

This work introduces Hellinger-UCB, a novel upper confidence bound strategy for stochastic multi-armed bandits that leverages the squared Hellinger distance to form confidence sets. The algorithm achieves optimal or near-optimal regret in one-parameter exponential-family settings and offers a closed-form solution under binomial rewards, enabling low-latency decisions in large-scale cold-start recommender systems. Theoretical analysis establishes an $O( ext{log}(T))$ regret bound, while numerical simulations and a real-world cold-start application demonstrate superior performance compared to KL-UCB and UCB1. The method provides practical benefits for content recommendation with new items, including faster learning and reduced computation, making it suitable for high-throughput production environments.

Abstract

In this paper, we study the stochastic multi-armed bandit problem, where the reward is driven by an unknown random variable. We propose a new variant of the Upper Confidence Bound (UCB) algorithm called Hellinger-UCB, which leverages the squared Hellinger distance to build the upper confidence bound. We prove that the Hellinger-UCB reaches the theoretical lower bound. We also show that the Hellinger-UCB has a solid statistical interpretation. We show that Hellinger-UCB is effective in finite time horizons with numerical experiments between Hellinger-UCB and other variants of the UCB algorithm. As a real-world example, we apply the Hellinger-UCB algorithm to solve the cold-start problem for a content recommender system of a financial app. With reasonable assumption, the Hellinger-UCB algorithm has a convenient but important lower latency feature. The online experiment also illustrates that the Hellinger-UCB outperforms both KL-UCB and UCB1 in the sense of a higher click-through rate (CTR).

HELLINGER-UCB: A novel algorithm for stochastic multi-armed bandit problem and cold start problem in recommender system

TL;DR

This work introduces Hellinger-UCB, a novel upper confidence bound strategy for stochastic multi-armed bandits that leverages the squared Hellinger distance to form confidence sets. The algorithm achieves optimal or near-optimal regret in one-parameter exponential-family settings and offers a closed-form solution under binomial rewards, enabling low-latency decisions in large-scale cold-start recommender systems. Theoretical analysis establishes an regret bound, while numerical simulations and a real-world cold-start application demonstrate superior performance compared to KL-UCB and UCB1. The method provides practical benefits for content recommendation with new items, including faster learning and reduced computation, making it suitable for high-throughput production environments.

Abstract

In this paper, we study the stochastic multi-armed bandit problem, where the reward is driven by an unknown random variable. We propose a new variant of the Upper Confidence Bound (UCB) algorithm called Hellinger-UCB, which leverages the squared Hellinger distance to build the upper confidence bound. We prove that the Hellinger-UCB reaches the theoretical lower bound. We also show that the Hellinger-UCB has a solid statistical interpretation. We show that Hellinger-UCB is effective in finite time horizons with numerical experiments between Hellinger-UCB and other variants of the UCB algorithm. As a real-world example, we apply the Hellinger-UCB algorithm to solve the cold-start problem for a content recommender system of a financial app. With reasonable assumption, the Hellinger-UCB algorithm has a convenient but important lower latency feature. The online experiment also illustrates that the Hellinger-UCB outperforms both KL-UCB and UCB1 in the sense of a higher click-through rate (CTR).
Paper Structure (22 sections, 8 theorems, 63 equations, 7 figures, 1 table, 1 algorithm)

This paper contains 22 sections, 8 theorems, 63 equations, 7 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Stochastic multi-armed bandit problem
  3. Cold start problem in recommender system
  4. Setup And Notations
  5. The Hellinger-UCB Algorithm
  6. Main algorithm
  7. Optimality of Hellinger-UCB
  8. NUMERICAL SIMULATION EXAMPLES
  9. Setting 1: Bernoulli rewards
  10. Setting 2: Poisson rewards
  11. Results interpolation
  12. Real world example: Article recommendation with cold start
  13. Binomial distribution assumption
  14. The advantage of Hellinger-UCB
  15. Solution of different UCBs
  16. ...and 7 more sections

Key Result

Theorem 1.1

If a policy has regret $\bar{R}_{T}=o(T^{a})$ for all $a>0$ as $T\to0$, the number of draws up to time $t$, $N_{i}(t)$ of any sub-optimal arm $i$ is lower bounded Therefore, the regret is lower-bounded .

Figures (7)

  • Figure 1: Bernoulli: pseudo regret
  • Figure 2: Bernoulli: average pseudo regret
  • Figure 3: Bernoulli: last step box-plot
  • Figure 4: Poisson: pseudo regret
  • Figure 5: Poisson: average pseudo regret
  • ...and 2 more figures

Theorems & Definitions (15)

  • Theorem 1.1
  • Theorem 3.1
  • proof
  • Theorem 3.2
  • proof
  • Lemma B.1
  • proof
  • Lemma B.2
  • proof
  • Lemma B.3
  • ...and 5 more