Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment

Qizhang Feng, Siva Rajesh Kasa, Santhosh Kumar Kasa, Hyokun Yun, Choon Hui Teo, Sravan Babu Bodapati

TL;DR

The paper investigates privacy risks in LLM alignment data by analyzing membership inference attacks (MIAs) on PPO- and DPO-based methods. It theoretically demonstrates that DPO overfits preference data and is more susceptible to MIAs, and then introduces PREMIA, a reference-based attack framework tailored to preference tuples. Empirically, PREMIA and existing MIAs reveal stronger MIA vulnerability for DPO across multiple models and tasks, with model size and task difficulty modulating the risk. The work highlights a practical privacy-utility trade-off in alignment strategies and motivates future privacy-preserving approaches for preference-based fine-tuning of LLMs.

Abstract

Large Language Models (LLMs) have seen widespread adoption due to their remarkable natural language capabilities. However, when deploying them in real-world settings, it is important to align LLMs to generate texts according to acceptable human standards. Methods such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) have enabled significant progress in refining LLMs using human preference data. However, the privacy concerns inherent in utilizing such preference data have yet to be adequately studied. In this paper, we investigate the vulnerability of LLMs aligned using two widely used methods - DPO and PPO - to membership inference attacks (MIAs). Our study has two main contributions: first, we theoretically motivate that DPO models are more vulnerable to MIA compared to PPO models; second, we introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (\uline{Pre}ference data \uline{MIA}). Using PREMIA and existing baselines we empirically show that DPO models have a relatively heightened vulnerability towards MIA.

Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment

TL;DR

The paper investigates privacy risks in LLM alignment data by analyzing membership inference attacks (MIAs) on PPO- and DPO-based methods. It theoretically demonstrates that DPO overfits preference data and is more susceptible to MIAs, and then introduces PREMIA, a reference-based attack framework tailored to preference tuples. Empirically, PREMIA and existing MIAs reveal stronger MIA vulnerability for DPO across multiple models and tasks, with model size and task difficulty modulating the risk. The work highlights a practical privacy-utility trade-off in alignment strategies and motivates future privacy-preserving approaches for preference-based fine-tuning of LLMs.

Abstract

Large Language Models (LLMs) have seen widespread adoption due to their remarkable natural language capabilities. However, when deploying them in real-world settings, it is important to align LLMs to generate texts according to acceptable human standards. Methods such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) have enabled significant progress in refining LLMs using human preference data. However, the privacy concerns inherent in utilizing such preference data have yet to be adequately studied. In this paper, we investigate the vulnerability of LLMs aligned using two widely used methods - DPO and PPO - to membership inference attacks (MIAs). Our study has two main contributions: first, we theoretically motivate that DPO models are more vulnerable to MIA compared to PPO models; second, we introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (\uline{Pre}ference data \uline{MIA}). Using PREMIA and existing baselines we empirically show that DPO models have a relatively heightened vulnerability towards MIA.
Paper Structure (42 sections, 7 theorems, 29 equations, 7 figures, 5 tables)

This paper contains 42 sections, 7 theorems, 29 equations, 7 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. PRELIMINARIES
  3. Model Alignment
  4. Proximal Policy Optimization (PPO)
  5. Direct Preference Optimization (DPO)
  6. Membership Inference Attacks (MIA) on LLMs
  7. Problem Statement
  8. Theoretical Results
  9. PREMIA FRAMEWORK
  10. For Individual Response
  11. For the Entire Preference Tuple
  12. EXPERIMENTS
  13. Research Questions
  14. Setup
  15. Models.
  16. ...and 27 more sections

Key Result

Lemma 1

Let $\pi_{DPO}$ be the policy obtained by optimizing Equation eq:DPO. Then, a) the corresponding implicit rewards on the preference pairs that optimize the BT model loss in Equation eq:RW_model are given by $r_d(x,y_w) = \beta \log \frac{\pi_{DPO}(y_w|x)}{\pi_{ref}(y_w|x)}$ and $r_d(x,y_l) = \beta \

Figures (7)

  • Figure 1: Overview of PREMIA framework for individual prompt-response pairs and the entire preference tuple
  • Figure 2: AUROC scores for $\text{MIA}_\text{Pair}$ detection for SE dataset
  • Figure 3: Train/Eval Aaccuracy for Mistral-7B on IMDB.
  • Figure 4: The standard 2-dimensional simplex forms a standard 3-dimensional tetrahedron with the origin $A$ whose volume is given by $\frac{1}{3!}$. The smaller tetrahedron $A'B'C'D$ has a volume $\frac{(1-\varepsilon)^3}{3!}$
  • Figure 5: The lower bound given here is much tighter compared to that of aubinais2023fundamental
  • ...and 2 more figures

Theorems & Definitions (11)

  • Lemma 1
  • proof
  • Remark 1
  • Proposition 1
  • Proposition 2
  • Lemma 2
  • Proposition 3
  • Definition 1
  • Lemma 3
  • Theorem 2.1
  • ...and 1 more