Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert

TL;DR

This work formalizes the risk of input-only PII memorization in fine-tuned LLMs and introduces a true-prefix probing method to quantify leakage under a realistic black-box threat. It benchmarks four privacy-preserving approaches—Differential Privacy, UnDial (machine unlearning), Regularization, and Direct Preference Optimization (DPO)—across synthetic multilingual and German medical datasets. The findings show post-training methods generally offer more robust privacy-utility trade-offs, while differential privacy provides strong leakage reductions in some settings but can introduce training instability, and no approach completely eliminates leakage. The study highlights persistent memorization risks in fine-tuned LLMs and advocates for scalable, robust defenses to enable privacy-preserving deployment in sensitive domains.

Abstract

Fine-tuning Large Language Models (LLMs) on sensitive datasets carries a substantial risk of unintended memorization and leakage of Personally Identifiable Information (PII), which can violate privacy regulations and compromise individual safety. In this work, we systematically investigate a critical and underexplored vulnerability: the exposure of PII that appears only in model inputs, not in training targets. Using both synthetic and real-world datasets, we design controlled extraction probes to quantify unintended PII memorization and study how factors such as language, PII frequency, task type, and model size influence memorization behavior. We further benchmark four privacy-preserving approaches including differential privacy, machine unlearning, regularization, and preference alignment, evaluating their trade-offs between privacy and task performance. Our results show that post-training methods generally provide more consistent privacy-utility trade-offs, while differential privacy achieves strong reduction in leakage in specific settings, although it can introduce training instability. These findings highlight the persistent challenge of memorization in fine-tuned LLMs and emphasize the need for robust, scalable privacy-preserving techniques.

Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

TL;DR

This work formalizes the risk of input-only PII memorization in fine-tuned LLMs and introduces a true-prefix probing method to quantify leakage under a realistic black-box threat. It benchmarks four privacy-preserving approaches—Differential Privacy, UnDial (machine unlearning), Regularization, and Direct Preference Optimization (DPO)—across synthetic multilingual and German medical datasets. The findings show post-training methods generally offer more robust privacy-utility trade-offs, while differential privacy provides strong leakage reductions in some settings but can introduce training instability, and no approach completely eliminates leakage. The study highlights persistent memorization risks in fine-tuned LLMs and advocates for scalable, robust defenses to enable privacy-preserving deployment in sensitive domains.

Abstract

Fine-tuning Large Language Models (LLMs) on sensitive datasets carries a substantial risk of unintended memorization and leakage of Personally Identifiable Information (PII), which can violate privacy regulations and compromise individual safety. In this work, we systematically investigate a critical and underexplored vulnerability: the exposure of PII that appears only in model inputs, not in training targets. Using both synthetic and real-world datasets, we design controlled extraction probes to quantify unintended PII memorization and study how factors such as language, PII frequency, task type, and model size influence memorization behavior. We further benchmark four privacy-preserving approaches including differential privacy, machine unlearning, regularization, and preference alignment, evaluating their trade-offs between privacy and task performance. Our results show that post-training methods generally provide more consistent privacy-utility trade-offs, while differential privacy achieves strong reduction in leakage in specific settings, although it can introduce training instability. These findings highlight the persistent challenge of memorization in fine-tuned LLMs and emphasize the need for robust, scalable privacy-preserving techniques.
Paper Structure (46 sections, 1 equation, 10 figures, 13 tables)

This paper contains 46 sections, 1 equation, 10 figures, 13 tables.

Table of Contents

  1. Introduction and Related Work
  2. Methodology
  3. Unintended, Input-only PII Memorization
  4. Mitigating memorization
  5. Experiments
  6. Datasets
  7. Privacy-preserving training
  8. Supervised Fine-Tuning
  9. Differential Privacy Fine-Tuning
  10. UnDial
  11. Regularization
  12. DPO
  13. Evaluation
  14. Results
  15. Understanding input-only PII memorization
  16. ...and 31 more sections

Figures (10)

  • Figure 1: Overview of our experiment setup depicting the unintended PII memorization scenario, our attack, and fine-tuning approaches.
  • Figure 2: Distribution of per‐token log‑likelihoods for ground‑truth PII completions.
  • Figure 3: Relation of extracted PII counts (enhanced TPA, Cross-memorization) and true counts in the DS dataset for Llama 3.2 1B (seed 2431).
  • Figure 4: PII extraction success ratio across languages for GretelAI-Financial with Llama 3.2 1B. Error bars represent 95% confidence intervals over 3 random seeds.
  • Figure 5: Effect of different model architectures and sizes on task performance and PII memorization on the Discharge Summary dataset. See \ref{['tab:results_llama_gemma_qwen']} for the complete breakdown of PII extraction results.
  • ...and 5 more figures