Membership Inference Attacks on LLM-based Recommender Systems

TL;DR

The paper addresses privacy risks in LLM-based recommender systems that use in-context prompts embedding user-history data. It proposes five membership inference attacks—Direct Inquiry, Contrast, Hallucination, Similarity, and Poisoning—designed for black-box ICL-LLM RecsSys and evaluates them across four LLMs and three datasets, showing that Direct Inquiry, Contrast, and Poisoning are highly effective, while Hallucination is largely ineffective. The results quantify attack advantages and F1-scores, and identify factors such as the number of prompts and the victim's position that influence attack success. The work highlights concrete privacy vulnerabilities in prompt-based customization and discusses potential defenses, including monitoring, prompt alignment, and differential privacy, underscoring the need for mitigations as ICL-LLM RecSys deployments scale.

Abstract

Large language models (LLMs) based Recommender Systems (RecSys) can flexibly adapt recommendation systems to different domains. It utilizes in-context learning (ICL), i.e., the prompts, to customize the recommendation functions, which include sensitive historical user-specific item interactions, e.g., implicit feedback like clicked items or explicit product reviews. Such private information may be exposed to novel privacy attack. However, no study has been done on this important issue. We design four membership inference attacks (MIAs), aiming to reveal whether victims' historical interactions have been used by system prompts. They are \emph{direct inquiry, hallucination, similarity, and poisoning attacks}, each of which utilizes the unique features of LLMs or RecSys. We have carefully evaluated them on three LLMs that have been used to develop ICL-LLM RecSys and two well-known RecSys benchmark datasets. The results confirm that the MIA threat on LLM RecSys is realistic: direct inquiry and poisoning attacks showing significantly high attack advantages. We have also analyzed the factors affecting these attacks, such as the number of shots in system prompts and the position of the victim in the shots.

Figures (15)

• Figure 1: Prompting Example for LLM RecSys
• Figure 2: System Architecture for ICL-RecSys
• Figure 3: The direct inquiry attack
• Figure 4: The contrast attack.
• Figure 5: The poisoning attack: the example in the user prompt tries to override the example in the system prompt.