Memorization of Named Entities in Fine-tuned BERT Models

TL;DR

This work addresses privacy risks in fine-tuned BERT by quantifying named-entity memorization when the model is trained on private data. It compares three fine-tuning regimes—Full, Partial, and Differentially Private (DP) with two prompts (naive and informed)—on Enron and Blog Authorship datasets, using a text-generation based extraction pipeline and NER to detect memorized entities, including a $k$-eidetic analysis. Results show memorization of private entities remains below 10% across setups; pre-trained BERT baselines exhibit similar leakage, while DP-fine-tuning dramatically reduces memorization at the cost of downstream accuracy, with reported privacy budgets of $\epsilon=9.79$ and $\epsilon=7.38$ for the two datasets. The findings suggest DP can be a viable privacy-preserving approach for BERT in scenarios where text generation is not critical, and point to future work evaluating other models and resilience against membership inference attacks.

Abstract

Privacy preserving deep learning is an emerging field in machine learning that aims to mitigate the privacy risks in the use of deep neural networks. One such risk is training data extraction from language models that have been trained on datasets, which contain personal and privacy sensitive information. In our study, we investigate the extent of named entity memorization in fine-tuned BERT models. We use single-label text classification as representative downstream task and employ three different fine-tuning setups in our experiments, including one with Differential Privacy (DP). We create a large number of text samples from the fine-tuned BERT models utilizing a custom sequential sampling strategy with two prompting strategies. We search in these samples for named entities and check if they are also present in the fine-tuning datasets. We experiment with two benchmark datasets in the domains of emails and blogs. We show that the application of DP has a detrimental effect on the text generation capabilities of BERT. Furthermore, we show that a fine-tuned BERT does not generate more named entities specific to the fine-tuning dataset than a BERT model that is pre-trained only. This suggests that BERT is unlikely to emit personal or privacy sensitive named entities. Overall, our results are important to understand to what extent BERT-based services are prone to training data extraction attacks.

Figures (5)

• Figure 1: An illustration of our framework for extracting training data entities from BERT. First, we fine-tune a pre-trained BERT on a private dataset. Next, we generate text samples from the fine-tuned model using prompts. Finally, we search the generated samples for the named entities that occur in the private dataset.
• Figure 2: An illustration of the different fine-tuning methods.
• Figure 3: The percentages of all entities successfully extracted from the models, compared by prompting methods.
• Figure 4: The percentages of private entities and private $1$-eidetic entities successfully extracted from the models with the use of naive prompting.
• Figure 5: Label distribution of the datasets