PCL: Prompt-based Continual Learning for User Modeling in Recommender Systems

TL;DR

The paper tackles the challenge of building generalized user representations across sequential downstream tasks in recommender systems without retraining on all tasks. It introduces PCL, which freezes a pretrained backbone and item embeddings while learning per-task position-wise prompts and task-informed contextual prompts to mitigate catastrophic forgetting and exploit inter-task relations. Through pretraining on an initial self-supervised task and subsequent prompt tuning, PCL demonstrates strong performance gains, robustness to task order, and the ability to produce universal user representations for unseen tasks. The findings highlight the practicality of prompt-based continual learning for scalable, adaptable user modeling in real-world recommender systems.

Abstract

User modeling in large e-commerce platforms aims to optimize user experiences by incorporating various customer activities. Traditional models targeting a single task often focus on specific business metrics, neglecting the comprehensive user behavior, and thus limiting their effectiveness. To develop more generalized user representations, some existing work adopts Multi-task Learning (MTL)approaches. But they all face the challenges of optimization imbalance and inefficiency in adapting to new tasks. Continual Learning (CL), which allows models to learn new tasks incrementally and independently, has emerged as a solution to MTL's limitations. However, CL faces the challenge of catastrophic forgetting, where previously learned knowledge is lost when the model is learning the new task. Inspired by the success of prompt tuning in Pretrained Language Models (PLMs), we propose PCL, a Prompt-based Continual Learning framework for user modeling, which utilizes position-wise prompts as external memory for each task, preserving knowledge and mitigating catastrophic forgetting. Additionally, we design contextual prompts to capture and leverage inter-task relationships during prompt tuning. We conduct extensive experiments on real-world datasets to demonstrate PCL's effectiveness.

Figures (4)

• Figure 1: The overall framework of PCL. The backbone model and item embeddings are frozen in the downstream tasks.
• Figure 2: Performance of each task w.r.t the task order.
• Figure 3: Performance w.r.t. the number of training epochs with $50\%$ cold-start items in Tenrec dataset when $lr=0.01$.
• Figure 4: Performance and convergence time of an MLP model on $T_4$ in Tenrec with user features obtained from different pretraining methods.