LIBER: Lifelong User Behavior Modeling Based on Large Language Models

TL;DR

This work proposes Lifelong User Behavior Modeling (LIBER) based on large language models, which includes three modules: (1) User Behavior Streaming Partition (UBSP), (2) User Interest Learning (UIL), and (3) User Interest Fusion (UIF).

Abstract

CTR prediction plays a vital role in recommender systems. Recently, large language models (LLMs) have been applied in recommender systems due to their emergence abilities. While leveraging semantic information from LLMs has shown some improvements in the performance of recommender systems, two notable limitations persist in these studies. First, LLM-enhanced recommender systems encounter challenges in extracting valuable information from lifelong user behavior sequences within textual contexts for recommendation tasks. Second, the inherent variability in human behaviors leads to a constant stream of new behaviors and irregularly fluctuating user interests. This characteristic imposes two significant challenges on existing models. On the one hand, it presents difficulties for LLMs in effectively capturing the dynamic shifts in user interests within these sequences, and on the other hand, there exists the issue of substantial computational overhead if the LLMs necessitate recurrent calls upon each update to the user sequences. In this work, we propose Lifelong User Behavior Modeling (LIBER) based on large language models, which includes three modules: (1) User Behavior Streaming Partition (UBSP), (2) User Interest Learning (UIL), and (3) User Interest Fusion (UIF). Initially, UBSP is employed to condense lengthy user behavior sequences into shorter partitions in an incremental paradigm, facilitating more efficient processing. Subsequently, UIL leverages LLMs in a cascading way to infer insights from these partitions. Finally, UIF integrates the textual outputs generated by the aforementioned processes to construct a comprehensive representation, which can be incorporated by any recommendation model to enhance performance. LIBER has been deployed on Huawei's music recommendation service and achieved substantial improvements in users' play count and play time by 3.01% and 7.69%.

Figures (4)

• Figure 1: The illustration of the lifelong user behavior incomprehension problem for LLMs. We report the AUC performance of DIEN and DIEN+Llama2 with different history lengths on Movielens-100k and Amazon-book datasets. DIEN is a traditional recommendation model that only utilizes ID-based information, whereas DIEN+Llama2 is an LLM-enhanced model and utilizes both ID-based and textual information, which first uses Llama2 to exploit the textual information of behavior sequences and then employs the output of Llama2 as an additional feature for DIEN. It is observed that for both two datasets, as the length of behavior sequence $K$ grows, the performances of DIEN are improved but the performances of DIEN+Llama2 decrease significantly.
• Figure 2: The overview architecture of LIBER. LIBER consists of three modules: User Behavior Streaming Partition Module, User Interest Learning Module, and User Interest Fusion Module.
• Figure 3: Example prompts for LIBER. The yellow, green, and blue text bubbles represent the prompt template, the content to be filled in the template, and the response generated by LLMs respectively (some text has been omitted for page limitations).
• Figure 4: Ablation study about the effectiveness of different components in LIBER.