Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPAR: Personalized Content-Based Recommendation via Long Engagement Attention

Chiyu Zhang, Yifei Sun, Jun Chen, Jie Lei, Muhammad Abdul-Mageed, Sinong Wang, Rong Jin, Sem Park, Ning Yao, Bo Long

TL;DR

SPAR addresses the challenge of leverging long user engagement histories for content-based recommendations by combining session-based PLM encoding with sparse poly-attention, an LLM-driven global user-interest profiler, and standalone user/item embeddings. The framework performs post-fusion processing to distill fine-grained token signals, yielding multiple embeddings per user and per candidate content, which are then interactively matched via a lightweight predictor trained with NCE loss. Empirical results on MIND-small and Goodreads show state-of-the-art performance across AUC and ranking metrics, with ablations confirming the importance of each component (session grouping, sparse attention, UIE, CCS, and LLM summaries). SPAR’s design offers robust, scalable retrieval-ready representations and demonstrates practical impact for large-scale, text-centric recommendation systems, while highlighting avenues for extending to multimodal features and addressing potential biases in LLM-generated profiling.

Abstract

Leveraging users' long engagement histories is essential for personalized content recommendations. The success of pretrained language models (PLMs) in NLP has led to their use in encoding user histories and candidate items, framing content recommendations as textual semantic matching tasks. However, existing works still struggle with processing very long user historical text and insufficient user-item interaction. In this paper, we introduce a content-based recommendation framework, SPAR, which effectively tackles the challenges of holistic user interest extraction from the long user engagement history. It achieves so by leveraging PLM, poly-attention layers and attention sparsity mechanisms to encode user's history in a session-based manner. The user and item side features are sufficiently fused for engagement prediction while maintaining standalone representations for both sides, which is efficient for practical model deployment. Moreover, we enhance user profiling by exploiting large language model (LLM) to extract global interests from user engagement history. Extensive experiments on two benchmark datasets demonstrate that our framework outperforms existing state-of-the-art (SoTA) methods.

SPAR: Personalized Content-Based Recommendation via Long Engagement Attention

TL;DR

SPAR addresses the challenge of leverging long user engagement histories for content-based recommendations by combining session-based PLM encoding with sparse poly-attention, an LLM-driven global user-interest profiler, and standalone user/item embeddings. The framework performs post-fusion processing to distill fine-grained token signals, yielding multiple embeddings per user and per candidate content, which are then interactively matched via a lightweight predictor trained with NCE loss. Empirical results on MIND-small and Goodreads show state-of-the-art performance across AUC and ranking metrics, with ablations confirming the importance of each component (session grouping, sparse attention, UIE, CCS, and LLM summaries). SPAR’s design offers robust, scalable retrieval-ready representations and demonstrates practical impact for large-scale, text-centric recommendation systems, while highlighting avenues for extending to multimodal features and addressing potential biases in LLM-generated profiling.

Abstract

Leveraging users' long engagement histories is essential for personalized content recommendations. The success of pretrained language models (PLMs) in NLP has led to their use in encoding user histories and candidate items, framing content recommendations as textual semantic matching tasks. However, existing works still struggle with processing very long user historical text and insufficient user-item interaction. In this paper, we introduce a content-based recommendation framework, SPAR, which effectively tackles the challenges of holistic user interest extraction from the long user engagement history. It achieves so by leveraging PLM, poly-attention layers and attention sparsity mechanisms to encode user's history in a session-based manner. The user and item side features are sufficiently fused for engagement prediction while maintaining standalone representations for both sides, which is efficient for practical model deployment. Moreover, we enhance user profiling by exploiting large language model (LLM) to extract global interests from user engagement history. Extensive experiments on two benchmark datasets demonstrate that our framework outperforms existing state-of-the-art (SoTA) methods.
Paper Structure (26 sections, 6 equations, 4 figures, 11 tables)

This paper contains 26 sections, 6 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. User History Encoder
  6. Session-Based PLM Encoding
  7. LLM User Interest Summary
  8. User History Summarizing (UHS)
  9. Sparse Attention in UHS
  10. User Interests Extracting (UIE)
  11. Candidate Content Encoder
  12. Candidate Content Summarizing (CCS)
  13. Engagement Predictor
  14. Experiments
  15. Results
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Overview of our framework, SPAR.
  • Figure 2: Influence of different hyperparameters.
  • Figure 3: Effects of user engagement history length.
  • Figure 4: Example of prompting an LLM as a user profiler. The text within the red box represents the input to the LLM, while the text within the green box is the LLM's generated output. In the input text, the part highlighted in orange signifies the task instruction, and the text in blue denotes the user's browsed news history.