Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond the Sequence: Statistics-Driven Pre-training for Stabilizing Sequential Recommendation Model

Sirui Wang, Peiguang Li, Yunsen Xian, Hongzhi Zhang

TL;DR

The paper tackles instability in sequential recommendation caused by random user actions. It introduces STDP, a statistics-driven pre-training framework that leverages item co-occurrence and attribute frequency to create robust supervision via CIP, PSS, FAP, and IAP tasks on a SASRec backbone. The joint pre-training objective L_pre combined with a standard NIP fine-tuning objective L_nip yields substantial performance gains across six datasets and demonstrates generalization to another model, GRU4Rec. This work highlights the value of incorporating stable statistical signals to mitigate noise and stabilize optimization in sequential recommendation systems.

Abstract

The sequential recommendation task aims to predict the item that user is interested in according to his/her historical action sequence. However, inevitable random action, i.e. user randomly accesses an item among multiple candidates or clicks several items at random order, cause the sequence fails to provide stable and high-quality signals. To alleviate the issue, we propose the StatisTics-Driven Pre-traing framework (called STDP briefly). The main idea of the work lies in the exploration of utilizing the statistics information along with the pre-training paradigm to stabilize the optimization of recommendation model. Specifically, we derive two types of statistical information: item co-occurrence across sequence and attribute frequency within the sequence. And we design the following pre-training tasks: 1) The co-occurred items prediction task, which encourages the model to distribute its attention on multiple suitable targets instead of just focusing on the next item that may be unstable. 2) We generate a paired sequence by replacing items with their co-occurred items and enforce its representation close with the original one, thus enhancing the model's robustness to the random noise. 3) To reduce the impact of random on user's long-term preferences, we encourage the model to capture sequence-level frequent attributes. The significant improvement over six datasets demonstrates the effectiveness and superiority of the proposal, and further analysis verified the generalization of the STDP framework on other models.

Beyond the Sequence: Statistics-Driven Pre-training for Stabilizing Sequential Recommendation Model

TL;DR

The paper tackles instability in sequential recommendation caused by random user actions. It introduces STDP, a statistics-driven pre-training framework that leverages item co-occurrence and attribute frequency to create robust supervision via CIP, PSS, FAP, and IAP tasks on a SASRec backbone. The joint pre-training objective L_pre combined with a standard NIP fine-tuning objective L_nip yields substantial performance gains across six datasets and demonstrates generalization to another model, GRU4Rec. This work highlights the value of incorporating stable statistical signals to mitigate noise and stabilize optimization in sequential recommendation systems.

Abstract

The sequential recommendation task aims to predict the item that user is interested in according to his/her historical action sequence. However, inevitable random action, i.e. user randomly accesses an item among multiple candidates or clicks several items at random order, cause the sequence fails to provide stable and high-quality signals. To alleviate the issue, we propose the StatisTics-Driven Pre-traing framework (called STDP briefly). The main idea of the work lies in the exploration of utilizing the statistics information along with the pre-training paradigm to stabilize the optimization of recommendation model. Specifically, we derive two types of statistical information: item co-occurrence across sequence and attribute frequency within the sequence. And we design the following pre-training tasks: 1) The co-occurred items prediction task, which encourages the model to distribute its attention on multiple suitable targets instead of just focusing on the next item that may be unstable. 2) We generate a paired sequence by replacing items with their co-occurred items and enforce its representation close with the original one, thus enhancing the model's robustness to the random noise. 3) To reduce the impact of random on user's long-term preferences, we encourage the model to capture sequence-level frequent attributes. The significant improvement over six datasets demonstrates the effectiveness and superiority of the proposal, and further analysis verified the generalization of the STDP framework on other models.
Paper Structure (18 sections, 9 equations, 2 figures, 3 tables)

This paper contains 18 sections, 9 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Approach
  5. Overview.
  6. Base Method.
  7. Statistics-driven pre-training tasks.
  8. Optimization.
  9. Experiments
  10. Experimental Settings
  11. Datasets.
  12. Implementation and Evaluation.
  13. Baseline Methods.
  14. Experimental Results
  15. Overall Performance.
  16. ...and 3 more sections

Figures (2)

  • Figure 1: The structure of StatisTics-Driven Pre-traing (STDP) framework. Bottom panel illustrates examples of statistics and how a paired sequence is created, then a base model is employed to model the original sequence and the paired sequence separately. The top panel illustrates pre-training tasks: Co-occurred Items Prediction (CIP), Paired Sequence Similarity (PSS), Frequent Attribute Prediction (FAP), and Item Attribute Prediction (IAP).
  • Figure 2: Performance comparison between GRU4Rec (blue), GRU4Rec with IAP (green), and GRU4Rec with STDP framework (red).