Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmbSum: Leveraging the Summarization Capabilities of Large Language Models for Content-Based Recommendations

Chiyu Zhang, Yifei Sun, Minghao Wu, Jun Chen, Jie Lei, Muhammad Abdul-Mageed, Rong Jin, Angli Liu, Ji Zhu, Sem Park, Ning Yao, Bo Long

TL;DR

EmbSum addresses the challenge of leveraging long user engagement histories for content-based recommendations by enabling offline pre-computations of user and item embeddings. It combines an encoder-decoder Transformer with poly-attention to produce User Poly-Embedding (UPE) and Content Poly-Embedding (CPE), while training a decoder to generate user-interest summaries supervised by large language models. The model optimizes a dual objective with NCE for CTR prediction and a summarization loss, achieving state-of-the-art results on the MIND and Goodreads datasets with a compact 61M-parameter backbone. Additionally, EmbSum provides human-readable user-interest summaries, enhancing explainability and practical usefulness for personalized content recommendation. Overall, EmbSum demonstrates that offline embedding, multi-embedding representations, and LLM-guided summarization can jointly improve accuracy and efficiency across domains while reducing model size.

Abstract

Content-based recommendation systems play a crucial role in delivering personalized content to users in the digital world. In this work, we introduce EmbSum, a novel framework that enables offline pre-computations of users and candidate items while capturing the interactions within the user engagement history. By utilizing the pretrained encoder-decoder model and poly-attention layers, EmbSum derives User Poly-Embedding (UPE) and Content Poly-Embedding (CPE) to calculate relevance scores between users and candidate items. EmbSum actively learns the long user engagement histories by generating user-interest summary with supervision from large language model (LLM). The effectiveness of EmbSum is validated on two datasets from different domains, surpassing state-of-the-art (SoTA) methods with higher accuracy and fewer parameters. Additionally, the model's ability to generate summaries of user interests serves as a valuable by-product, enhancing its usefulness for personalized content recommendations.

EmbSum: Leveraging the Summarization Capabilities of Large Language Models for Content-Based Recommendations

TL;DR

EmbSum addresses the challenge of leveraging long user engagement histories for content-based recommendations by enabling offline pre-computations of user and item embeddings. It combines an encoder-decoder Transformer with poly-attention to produce User Poly-Embedding (UPE) and Content Poly-Embedding (CPE), while training a decoder to generate user-interest summaries supervised by large language models. The model optimizes a dual objective with NCE for CTR prediction and a summarization loss, achieving state-of-the-art results on the MIND and Goodreads datasets with a compact 61M-parameter backbone. Additionally, EmbSum provides human-readable user-interest summaries, enhancing explainability and practical usefulness for personalized content recommendation. Overall, EmbSum demonstrates that offline embedding, multi-embedding representations, and LLM-guided summarization can jointly improve accuracy and efficiency across domains while reducing model size.

Abstract

Content-based recommendation systems play a crucial role in delivering personalized content to users in the digital world. In this work, we introduce EmbSum, a novel framework that enables offline pre-computations of users and candidate items while capturing the interactions within the user engagement history. By utilizing the pretrained encoder-decoder model and poly-attention layers, EmbSum derives User Poly-Embedding (UPE) and Content Poly-Embedding (CPE) to calculate relevance scores between users and candidate items. EmbSum actively learns the long user engagement histories by generating user-interest summary with supervision from large language model (LLM). The effectiveness of EmbSum is validated on two datasets from different domains, surpassing state-of-the-art (SoTA) methods with higher accuracy and fewer parameters. Additionally, the model's ability to generate summaries of user interests serves as a valuable by-product, enhancing its usefulness for personalized content recommendations.
Paper Structure (29 sections, 6 equations, 3 figures, 5 tables)

This paper contains 29 sections, 6 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation
  4. Overview of EmbSum
  5. User Engagement Modeling
  6. Session Encoding
  7. User Engagement Summarization
  8. User Poly-Embedding
  9. Candidate Content Modeling
  10. CTR Prediction and Training
  11. CTR Prediction
  12. Training
  13. Experiments
  14. Baselines
  15. Dataset
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Overview of our EmbSum framework. Note that the user summaries generated by LLMs are only used in training.
  • Figure 2: Illustration of using an LLM for user interest profiling. The input provided to the LLM is enclosed in a red box, and the output generated by the LLM is shown in a green box. The segment marked in orange within the input specifies the instruction for the task, whereas the portion in blue highlights the history of news browsed by the user.
  • Figure 3: Influence of different hyperparameters.