Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendation

TL;DR

DPCPL tackles the dual challenges of noise in multi-behavior sequential data and costly fine-tuning by introducing an FFT-based Efficient Behavior Miner for denoising and a Customized Prompt Learning module for parameter-efficient adaptation. The denoising stage reduces noise across multiple time scales with frequency-domain fusion and a Chunked Diagonal Mechanism to keep parameter counts manageable, while the prompting stage personalizes, progressively refines, and diversifies prompts across model layers using a Prompt Factor Gate and a compactness regularization. Empirical results on three large real-world datasets show consistent state-of-the-art performance with significantly reduced tuning costs, and ablations confirm the effectiveness of each component (denoising, personalization, progressiveness, and diversity). Overall, DPCPL offers a scalable, efficient pathway to high-precision multi-behavior recommendations in latency-constrained environments.

Abstract

In the realm of recommendation systems, users exhibit a diverse array of behaviors when interacting with items. This phenomenon has spurred research into learning the implicit semantic relationships between these behaviors to enhance recommendation performance. However, these methods often entail high computational complexity. To address concerns regarding efficiency, pre-training presents a viable solution. Its objective is to extract knowledge from extensive pre-training data and fine-tune the model for downstream tasks. Nevertheless, previous pre-training methods have primarily focused on single-behavior data, while multi-behavior data contains significant noise. Additionally, the fully fine-tuning strategy adopted by these methods still imposes a considerable computational burden. In response to this challenge, we propose DPCPL, the first pre-training and prompt-tuning paradigm tailored for Multi-Behavior Sequential Recommendation. Specifically, in the pre-training stage, we commence by proposing a novel Efficient Behavior Miner (EBM) to filter out the noise at multiple time scales, thereby facilitating the comprehension of the contextual semantics of multi-behavior sequences. Subsequently, we propose to tune the pre-trained model in a highly efficient manner with the proposed Customized Prompt Learning (CPL) module, which generates personalized, progressive, and diverse prompts to fully exploit the potential of the pre-trained model effectively. Extensive experiments on three real-world datasets have unequivocally demonstrated that DPCPL not only exhibits high efficiency and effectiveness, requiring minimal parameter adjustments but also surpasses the state-of-the-art performance across a diverse range of downstream tasks.

Figures (5)

• Figure 1: Illustration of our motivations: the rapid increase in user interactions leads to a substantial amount of data, which introduces significant noise. This noise presents challenges in accurately capturing user preferences and highlights the need for more robust pre-training and fine-tuning methods for efficient transfer to downstream tasks.
• Figure 2: Illustration of our problem: Our task involves taking a sequence of user behaviors as input and predicting the next item of user interaction under a specific target behavior. This target behavior can vary, serving as different downstream tasks.
• Figure 3: Our DPCPL model, a pre-training and prompt-tuning paradigm tailored for Multi-Behavior Sequential Recommendation. Specifically, in the pre-training stage, we commence by proposing a novel Efficient Behavior Miner (EBM) to filter out the noise at multiple time scales, therefore facilitating the comprehension of the contextual semantics of multi-behavior sequences. Subsequently, we propose to tune the pre-trained model in a highly efficient manner with the proposed Customized Prompt Learning (CPL) module, which generates personalized, progressive, and diverse prompts to fully exploit the potential of the pre-trained model effectively.
• Figure 4: Hyperparameter sensitivity analysis for the selected hyperparameters: number of prompt factors of each group ($N$), number of tokens in prompt ($C$), compactness regularity parameter ($\lambda$), and number of pre-trained model layers ($L$).
• Figure 5: Dimensionality reduction visualization of prompt vectors at each layer of models in the CIKM dataset, where the legend indicates the layer of the pre-trained model.