Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M

Dario Di Palma, Felice Antonio Merra, Maurizio Sfilio, Vito Walter Anelli, Fedelucio Narducci, Tommaso Di Noia

TL;DR

The paper investigates whether public recommendation data, specifically MovieLens-1M, is memorized by GPT and Llama LLMs and how this memorization affects recommendation quality. It defines memorization across items, users, and interactions, and introduces a prompt-based extraction framework with three coverage metrics to quantify memorization across model families and sizes. The findings show substantial memorization across content and a positive correlation between memorization and recommender performance, highlighting a significant leakage risk as larger models memorize more and reflect dataset biases, particularly popularity effects. The results call for caution in evaluating LLM-based recommenders and motivate future work on memorization mitigation and robust benchmarking methods to ensure reliable generalization across unseen datasets.

Abstract

Large Language Models (LLMs) have become increasingly central to recommendation scenarios due to their remarkable natural language understanding and generation capabilities. Although significant research has explored the use of LLMs for various recommendation tasks, little effort has been dedicated to verifying whether they have memorized public recommendation dataset as part of their training data. This is undesirable because memorization reduces the generalizability of research findings, as benchmarking on memorized datasets does not guarantee generalization to unseen datasets. Furthermore, memorization can amplify biases, for example, some popular items may be recommended more frequently than others. In this work, we investigate whether LLMs have memorized public recommendation datasets. Specifically, we examine two model families (GPT and Llama) across multiple sizes, focusing on one of the most widely used dataset in recommender systems: MovieLens-1M. First, we define dataset memorization as the extent to which item attributes, user profiles, and user-item interactions can be retrieved by prompting the LLMs. Second, we analyze the impact of memorization on recommendation performance. Lastly, we examine whether memorization varies across model families and model sizes. Our results reveal that all models exhibit some degree of memorization of MovieLens-1M, and that recommendation performance is related to the extent of memorization. We have made all the code publicly available at: https://github.com/sisinflab/LLM-MemoryInspector

Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M

TL;DR

The paper investigates whether public recommendation data, specifically MovieLens-1M, is memorized by GPT and Llama LLMs and how this memorization affects recommendation quality. It defines memorization across items, users, and interactions, and introduces a prompt-based extraction framework with three coverage metrics to quantify memorization across model families and sizes. The findings show substantial memorization across content and a positive correlation between memorization and recommender performance, highlighting a significant leakage risk as larger models memorize more and reflect dataset biases, particularly popularity effects. The results call for caution in evaluating LLM-based recommenders and motivate future work on memorization mitigation and robust benchmarking methods to ensure reliable generalization across unseen datasets.

Abstract

Large Language Models (LLMs) have become increasingly central to recommendation scenarios due to their remarkable natural language understanding and generation capabilities. Although significant research has explored the use of LLMs for various recommendation tasks, little effort has been dedicated to verifying whether they have memorized public recommendation dataset as part of their training data. This is undesirable because memorization reduces the generalizability of research findings, as benchmarking on memorized datasets does not guarantee generalization to unseen datasets. Furthermore, memorization can amplify biases, for example, some popular items may be recommended more frequently than others. In this work, we investigate whether LLMs have memorized public recommendation datasets. Specifically, we examine two model families (GPT and Llama) across multiple sizes, focusing on one of the most widely used dataset in recommender systems: MovieLens-1M. First, we define dataset memorization as the extent to which item attributes, user profiles, and user-item interactions can be retrieved by prompting the LLMs. Second, we analyze the impact of memorization on recommendation performance. Lastly, we examine whether memorization varies across model families and model sizes. Our results reveal that all models exhibit some degree of memorization of MovieLens-1M, and that recommendation performance is related to the extent of memorization. We have made all the code publicly available at: https://github.com/sisinflab/LLM-MemoryInspector
Paper Structure (10 sections, 4 equations, 4 figures, 2 tables)

This paper contains 10 sections, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Definitions of Memorization
  4. Evaluation Protocol
  5. Results and Discussion
  6. Analysis of Memorization
  7. Impact on Recommendation
  8. Impact of Model Scale
  9. Popularity Memorization
  10. Conclusion and open directions

Figures (4)

  • Figure 1: Few-Shot Prompting for Item/User Data Extraction
  • Figure 2: Few-Shot Prompting for Interactions Data Extraction
  • Figure 3: Zero-Shot Prompting for Recommendation Task
  • Figure 4: Comparison of item coverage across models by popularity tier. The figure shows the percentage of items covered in three categories: Highly Popular (Top 20%), Moderately Popular (Middle 20%), and Rarely Interacted (Bottom 20%).

Theorems & Definitions (4)

  • Definition 1: Item Memorization
  • Definition 2: User Memorization
  • Definition 3: (User-Item) Interaction Memorization
  • Definition 4: Memorization Coverage