Exploring Approaches for Detecting Memorization of Recommender System Data in Large Language Models

TL;DR

This study investigates how memorized data from MovieLens-1M can be detected in large language models used for recommender-style tasks. It compares three probing families—manual jailbroken prompting, unsupervised latent knowledge discovery (CCS and Cluster-Norm), and Automatic Prompt Engineering (APE)—across item, user, and rating fields using LLaMA-1B/3B. Key findings show that jailbreak prompts yield limited, inconsistent gains, unsupervised probes reliably detect textual item data but struggle with numerical fields, and APE provides the strongest item-level extraction with moderate success, while numerical data remain challenging due to tokenizer effects. Overall, automatically optimizing prompts emerges as the most promising strategy for revealing memorized samples, with implications for evaluating and mitigating data leakage in LLM-based recommender systems; future work should scale to larger models and broader datasets and refine approaches for numerical data.

Abstract

Large Language Models (LLMs) are increasingly applied in recommendation scenarios due to their strong natural language understanding and generation capabilities. However, they are trained on vast corpora whose contents are not publicly disclosed, raising concerns about data leakage. Recent work has shown that the MovieLens-1M dataset is memorized by both the LLaMA and OpenAI model families, but the extraction of such memorized data has so far relied exclusively on manual prompt engineering. In this paper, we pose three main questions: Is it possible to enhance manual prompting? Can LLM memorization be detected through methods beyond manual prompting? And can the detection of data leakage be automated? To address these questions, we evaluate three approaches: (i) jailbreak prompt engineering; (ii) unsupervised latent knowledge discovery, probing internal activations via Contrast-Consistent Search (CCS) and Cluster-Norm; and (iii) Automatic Prompt Engineering (APE), which frames prompt discovery as a meta-learning process that iteratively refines candidate instructions. Experiments on MovieLens-1M using LLaMA models show that jailbreak prompting does not improve the retrieval of memorized items and remains inconsistent; CCS reliably distinguishes genuine from fabricated movie titles but fails on numerical user and rating data; and APE retrieves item-level information with moderate success yet struggles to recover numerical interactions. These findings suggest that automatically optimizing prompts is the most promising strategy for extracting memorized samples.

Figures (4)

• Figure 1: Jailbreak prompt-engineering workflow: select dataset samples; craft a prompt with few-shot examples and chain-of-thought instructions incorporating jailbreaking prompt; query the LLM; manual analysis of responses for validity, duplication, and hallucinations; then iterate on the prompt.
• Figure 2: Context Compliance Attack (CCA) jailbreak prompt tailored for MovieLens– 1M memorization discovery. The fabricated history primes the model to output exact dataset fields when given keys.
• Figure 3: Unsupervised latent knowledge discovery via CCS. Positive and negative statements are processed by the LLM to extract hidden activations, which are paired and fed to a linear probe optimised to assign high scores to true statements and low scores to false ones.
• Figure 4: PCA visualizations of the dataset. (a) Item embeddings. (b) User embeddings.