Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Encoder-Only Transformers for Session-Based Recommendation Systems

Anis Redjdal, Luis Pinto, Michel Desmarais

TL;DR

This work introduces Sequential Masked Modeling, a novel approach for encoder-only transformer architectures to tackle the challenges of single-session recommendation that consistently outperform all models that rely on the same amount of information, while even rivaling methods that have access to more extensive user history.

Abstract

Session-based recommendation is the task of predicting the next item a user will interact with, often without access to historical user data. In this work, we introduce Sequential Masked Modeling, a novel approach for encoder-only transformer architectures to tackle the challenges of single-session recommendation. Our method combines data augmentation through window sliding with a unique penultimate token masking strategy to capture sequential dependencies more effectively. By enhancing how transformers handle session data, Sequential Masked Modeling significantly improves next-item prediction performance. We evaluate our approach on three widely-used datasets, Yoochoose 1/64, Diginetica, and Tmall, comparing it to state-of-the-art single-session, cross-session, and multi-relation approaches. The results demonstrate that our Transformer-SMM models consistently outperform all models that rely on the same amount of information, while even rivaling methods that have access to more extensive user history. This study highlights the potential of encoder-only transformers in session-based recommendation and opens the door for further improvements.

Optimizing Encoder-Only Transformers for Session-Based Recommendation Systems

TL;DR

This work introduces Sequential Masked Modeling, a novel approach for encoder-only transformer architectures to tackle the challenges of single-session recommendation that consistently outperform all models that rely on the same amount of information, while even rivaling methods that have access to more extensive user history.

Abstract

Session-based recommendation is the task of predicting the next item a user will interact with, often without access to historical user data. In this work, we introduce Sequential Masked Modeling, a novel approach for encoder-only transformer architectures to tackle the challenges of single-session recommendation. Our method combines data augmentation through window sliding with a unique penultimate token masking strategy to capture sequential dependencies more effectively. By enhancing how transformers handle session data, Sequential Masked Modeling significantly improves next-item prediction performance. We evaluate our approach on three widely-used datasets, Yoochoose 1/64, Diginetica, and Tmall, comparing it to state-of-the-art single-session, cross-session, and multi-relation approaches. The results demonstrate that our Transformer-SMM models consistently outperform all models that rely on the same amount of information, while even rivaling methods that have access to more extensive user history. This study highlights the potential of encoder-only transformers in session-based recommendation and opens the door for further improvements.

Paper Structure

This paper contains 36 sections, 6 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. Problem Statement
  5. Language Models
  6. Masked Language Modeling
  7. Causal Language Modeling
  8. The proposed method
  9. Data Augmentation with Window Sliding
  10. Masking the Penultimate Token
  11. Hypotheses
  12. Advantages of Over
  13. Advantages of Over
  14. Masking the penultimate instead of the last
  15. Optimizing techniques
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Next-click prediction task cloudera2021session.
  • Figure 2: SMM training representation. The [M] token is the masked item the model is trained to predict. Red tokens are cut from the input due to the maximum sequence length constraint ($max\_len = 5$), and blue tokens are the ones visible in the model's input sequence.
  • Figure 3: Performance impact of various enhancement techniques applied to the BERT architecture trained with the Sequential Masked Modeling objective.
  • Figure 4: Optimized BERT model architecture.