Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Bandit Algorithms with LLMs for Time-varying User Preferences in Streaming Recommendations

Chenglei Shen, Yi Zhan, Weijie Yu, Xiao Zhang, Jun Xu

TL;DR

This work tackles time-varying user preferences in streaming recommendations by formulating a time-aware contextual bandit framework. It introduces HyperBandit+, which integrates a time-conditioned hypernetwork to adapt to evolving user preferences, Euler embeddings for robust time encoding, and LLM Start to provide a warm-start via offline synthetic data. A low-rank factorization reduces training complexity, and rigorous dynamic regret analysis shows sublinear guarantees in non-stationary environments. Empirical results on real-world short-video and POI datasets demonstrate that HyperBandit+ consistently surpasses strong baselines, especially in early online stages, highlighting practical impact for real-time recommendations with evolving user tastes.

Abstract

In real-world streaming recommender systems, user preferences evolve dynamically over time. Existing bandit-based methods treat time merely as a timestamp, neglecting its explicit relationship with user preferences and leading to suboptimal performance. Moreover, online learning methods often suffer from inefficient exploration-exploitation during the early online phase. To address these issues, we propose HyperBandit+, a novel contextual bandit policy that integrates a time-aware hypernetwork to adapt to time-varying user preferences and employs a large language model-assisted warm-start mechanism (LLM Start) to enhance exploration-exploitation efficiency in the early online phase. Specifically, HyperBandit+ leverages a neural network that takes time features as input and generates parameters for estimating time-varying rewards by capturing the correlation between time and user preferences. Additionally, the LLM Start mechanism employs multi-step data augmentation to simulate realistic interaction data for effective offline learning, providing warm-start parameters for the bandit policy in the early online phase. To meet real-time streaming recommendation demands, we adopt low-rank factorization to reduce hypernetwork training complexity. Theoretically, we rigorously establish a sublinear regret upper bound that accounts for both the hypernetwork and the LLM warm-start mechanism. Extensive experiments on real-world datasets demonstrate that HyperBandit+ consistently outperforms state-of-the-art baselines in terms of accumulated rewards.

Enhancing Bandit Algorithms with LLMs for Time-varying User Preferences in Streaming Recommendations

TL;DR

This work tackles time-varying user preferences in streaming recommendations by formulating a time-aware contextual bandit framework. It introduces HyperBandit+, which integrates a time-conditioned hypernetwork to adapt to evolving user preferences, Euler embeddings for robust time encoding, and LLM Start to provide a warm-start via offline synthetic data. A low-rank factorization reduces training complexity, and rigorous dynamic regret analysis shows sublinear guarantees in non-stationary environments. Empirical results on real-world short-video and POI datasets demonstrate that HyperBandit+ consistently surpasses strong baselines, especially in early online stages, highlighting practical impact for real-time recommendations with evolving user tastes.

Abstract

In real-world streaming recommender systems, user preferences evolve dynamically over time. Existing bandit-based methods treat time merely as a timestamp, neglecting its explicit relationship with user preferences and leading to suboptimal performance. Moreover, online learning methods often suffer from inefficient exploration-exploitation during the early online phase. To address these issues, we propose HyperBandit+, a novel contextual bandit policy that integrates a time-aware hypernetwork to adapt to time-varying user preferences and employs a large language model-assisted warm-start mechanism (LLM Start) to enhance exploration-exploitation efficiency in the early online phase. Specifically, HyperBandit+ leverages a neural network that takes time features as input and generates parameters for estimating time-varying rewards by capturing the correlation between time and user preferences. Additionally, the LLM Start mechanism employs multi-step data augmentation to simulate realistic interaction data for effective offline learning, providing warm-start parameters for the bandit policy in the early online phase. To meet real-time streaming recommendation demands, we adopt low-rank factorization to reduce hypernetwork training complexity. Theoretically, we rigorously establish a sublinear regret upper bound that accounts for both the hypernetwork and the LLM warm-start mechanism. Extensive experiments on real-world datasets demonstrate that HyperBandit+ consistently outperforms state-of-the-art baselines in terms of accumulated rewards.
Paper Structure (44 sections, 4 theorems, 39 equations, 11 figures, 6 tables, 2 algorithms)

This paper contains 44 sections, 4 theorems, 39 equations, 11 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hypernetworks (HNs)
  4. Bandits in Non-stationary Environment
  5. LLM-enhanced Recommendation
  6. Problem Formulation
  7. Bandit-based Streaming Recommendation
  8. Time-Varying User Preferences
  9. HyperBandit+: The Proposed Algorithm
  10. Algorithm Overview
  11. Data Augmentation via LLMs
  12. Recommendation with Side Information
  13. LLM-Enhanced Embedding
  14. LLM Start
  15. Hypernetwork Assisted Bandit Policy
  16. ...and 29 more sections

Key Result

theorem 1

The ridge regression problem presented in Eq. eq:regularized_quadratic_loss_enhanced admits a closed-form solution, which is given as follows: where $\mathcal{U}_{a, t}$ denotes the set of users (possibly with duplicates) who have been recommended item $a$ until time $t$, and $\bm I_{l_a} \in \mathbb{R}^{l_a \times l_a}$ is a identity matrix. The statistics $\left(\bm \Psi_{a,t}, \bm b_{a,t}\righ

Figures (11)

  • Figure 1: Treemap visualizations illustrating periodic shifts in the Foursquare-NYC points-of-interest (POI) dataset for morning and night periods on weekdays and weekends. Block sizes correspond to the frequency of visits, with larger blocks indicating higher visit counts.
  • Figure 2: The illustrations of the data sparsity.
  • Figure 3: The structure of HyperBandit+ (HyperBandit with extensions of LLM Start, Euler Embedding and LLM-Enhanced Embedding at time $t$). The workflow involves the bandit policy selecting an arm (recommending an item) from the candidate pool, interacting with the environment to obtain feedback, and subsequently updating the bandit policy. "Stage One" and "Stage Two" represent the two environments that interact with the bandit policy sequentially. The symbol indicates the availability of the module during the online stage, while represents the offline policy.
  • Figure 4: Constructed prompts for LLMs' side information enhancing and data simulation. The light gray box contains role assignment, task description, specific information, and output format description. The dark gray box contains the corresponding format for the response.
  • Figure 5: The distribution of singular eigenvalues (SEs) of user preference matrices across different time periods. The horizontal axis represents the index of SEs, arranged in descending order, while the vertical axis represents the time periods. The darkness of the colors corresponds to the magnitude of the singular values.
  • ...and 6 more figures

Theorems & Definitions (4)

  • theorem 1: Closed-form Solution of Eq. \ref{['eq:regularized_quadratic_loss_enhanced']}
  • lemma 1
  • lemma 2
  • theorem 2: Dynamic Regret Bound with LLM Augmentation