Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modeling Attrition in Recommender Systems with Departing Bandits

Omer Ben-Porat, Lee Cohen, Liu Leqi, Zachary C. Lipton, Yishay Mansour

TL;DR

We introduce Departing Bandits, a multi-armed bandit variant where user horizons depend on the recommender's policy and where latent user types govern both click probabilities and departure risks via $P$ and $\Lambda$. When parameters are known, planning reduces to a POMDP with latent states; in the two-type, two-category setting the optimal policy is threshold-based and computable in constant time by analyzing three structural regimes of $P$. For unknown parameters, we develop a UCB-based framework for sub-exponential returns, achieving $\tilde{O}(\sqrt{T})$ regret in both the single-type and the two-type two-category cases by restricting to a compact policy class of threshold policies. These results enable principled online learning and planning for recommender systems where exploration can shorten user sessions, with potential practical impact on policy design under user churn.

Abstract

Traditionally, when recommender systems are formalized as multi-armed bandits, the policy of the recommender system influences the rewards accrued, but not the length of interaction. However, in real-world systems, dissatisfied users may depart (and never come back). In this work, we propose a novel multi-armed bandit setup that captures such policy-dependent horizons. Our setup consists of a finite set of user types, and multiple arms with Bernoulli payoffs. Each (user type, arm) tuple corresponds to an (unknown) reward probability. Each user's type is initially unknown and can only be inferred through their response to recommendations. Moreover, if a user is dissatisfied with their recommendation, they might depart the system. We first address the case where all users share the same type, demonstrating that a recent UCB-based algorithm is optimal. We then move forward to the more challenging case, where users are divided among two types. While naive approaches cannot handle this setting, we provide an efficient learning algorithm that achieves $\tilde{O}(\sqrt{T})$ regret, where $T$ is the number of users.

Modeling Attrition in Recommender Systems with Departing Bandits

TL;DR

We introduce Departing Bandits, a multi-armed bandit variant where user horizons depend on the recommender's policy and where latent user types govern both click probabilities and departure risks via and . When parameters are known, planning reduces to a POMDP with latent states; in the two-type, two-category setting the optimal policy is threshold-based and computable in constant time by analyzing three structural regimes of . For unknown parameters, we develop a UCB-based framework for sub-exponential returns, achieving regret in both the single-type and the two-type two-category cases by restricting to a compact policy class of threshold policies. These results enable principled online learning and planning for recommender systems where exploration can shorten user sessions, with potential practical impact on policy design under user churn.

Abstract

Traditionally, when recommender systems are formalized as multi-armed bandits, the policy of the recommender system influences the rewards accrued, but not the length of interaction. However, in real-world systems, dissatisfied users may depart (and never come back). In this work, we propose a novel multi-armed bandit setup that captures such policy-dependent horizons. Our setup consists of a finite set of user types, and multiple arms with Bernoulli payoffs. Each (user type, arm) tuple corresponds to an (unknown) reward probability. Each user's type is initially unknown and can only be inferred through their response to recommendations. Moreover, if a user is dissatisfied with their recommendation, they might depart the system. We first address the case where all users share the same type, demonstrating that a recent UCB-based algorithm is optimal. We then move forward to the more challenging case, where users are divided among two types. While naive approaches cannot handle this setting, we provide an efficient learning algorithm that achieves regret, where is the number of users.
Paper Structure (25 sections, 27 theorems, 110 equations, 2 tables, 2 algorithms)

This paper contains 25 sections, 27 theorems, 110 equations, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Departing Bandits
  4. Example
  5. UCB Policy for Sub-exponential Returns
  6. Sub-exponential Returns
  7. Single User Type
  8. Two User Types and Two Categories
  9. Efficient Planning
  10. Characterizing the Optimal Policy
  11. Dominant Row
  12. Dominant Column
  13. Dominant Diagonal
  14. Learning: UCB-based regret bound
  15. Conclusions and Discussion
  16. ...and 10 more sections

Key Result

Theorem 3.2

Let $\Pi$ be a set of policies with the associated parameters $\tilde{\tau},\eta$. Let $\pi_1,\dots,\pi_T$ be the policies Algorithm alg-policy-UCB selects. It holds that

Theorems & Definitions (43)

  • Definition 3.1
  • Theorem 3.2
  • Lemma 4.0
  • Lemma 4.1
  • Theorem 4.2
  • Remark 5.1
  • Definition 5.2: Belief-category walk
  • Theorem 5.3
  • Lemma 5.3
  • Theorem 5.4
  • ...and 33 more