Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Data Privacy in Large Language Models through Private Association Editing

Davide Venditti, Elena Sofia Ruzzetti, Giancarlo A. Xompero, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, Fabio Massimo Zanzotto

TL;DR

Private Association Editing (PAE) is designed to effectively remove Personally Identifiable Information (PII) without retraining the model, and experimental results demonstrate the effectiveness of PAE with respect to alternative baseline methods.

Abstract

Large language models (LLMs) require a significant redesign in solutions to preserve privacy in data-intensive applications due to their text-generation capabilities. Indeed, LLMs tend to memorize and emit private information when maliciously prompted. In this paper, we introduce Private Association Editing (PAE) as a novel defense approach for private data leakage. PAE is designed to effectively remove Personally Identifiable Information (PII) without retraining the model. Experimental results demonstrate the effectiveness of PAE with respect to alternative baseline methods. We believe PAE will serve as a critical tool in the ongoing effort to protect data privacy in LLMs, encouraging the development of safer models for real-world applications.

Enhancing Data Privacy in Large Language Models through Private Association Editing

TL;DR

Private Association Editing (PAE) is designed to effectively remove Personally Identifiable Information (PII) without retraining the model, and experimental results demonstrate the effectiveness of PAE with respect to alternative baseline methods.

Abstract

Large language models (LLMs) require a significant redesign in solutions to preserve privacy in data-intensive applications due to their text-generation capabilities. Indeed, LLMs tend to memorize and emit private information when maliciously prompted. In this paper, we introduce Private Association Editing (PAE) as a novel defense approach for private data leakage. PAE is designed to effectively remove Personally Identifiable Information (PII) without retraining the model. Experimental results demonstrate the effectiveness of PAE with respect to alternative baseline methods. We believe PAE will serve as a critical tool in the ongoing effort to protect data privacy in LLMs, encouraging the development of safer models for real-world applications.

Paper Structure

This paper contains 27 sections, 4 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Attacking and Defending LLMs from Private Data Leakage with Private Association Editing
  4. Training Data Extraction Attacks to recover Sensitive Information
  5. Private Association Editing as Efficient Defense against Privacy Attacks
  6. PAE Cards
  7. PAE Update Strategy on Model's Weights
  8. Evaluating Language Modeling Performance
  9. Experiments
  10. Experimental Setup
  11. Analized LLM and related datasets
  12. PAE and baselines application
  13. Evaluation of post-edit LLM
  14. Results and Discussion
  15. LLMs leak Private Information
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Preserving privacy for LLMs by using Private Association Editing
  • Figure 2: Memorization Attack against sequentially post-edit models. The smaller the batch size $k$, the larger the number of sequential updates necessary to edit all the private email addresses leaked by the original model.
  • Figure 3: The post-edit model is increasingly different from the pre-edit model as $\lambda$ increases. As discussed in Section \ref{['sec:consistency']}, this is an indication of a diminished utility of the model.