Exploring prompts to elicit memorization in masked language model-based named entity recognition

TL;DR

This study probes how prompt design affects memorization detection in MLM-based NER systems trained on CoNLL-2003. It uses 400 prompts generated from a pairwise PER dataset sourced from CoNLL and Wikidata across 6 publicly available models, measuring memorization via the confidence gap between In-train and Out-train PERs, $C_k(PER)$. The results reveal up to a $16$ percentage-point difference between the best and worst prompts on the same model; prompt engineering yields small but tangible gains, while ensemble methods offer limited benefit. Overall, prompt effectiveness is model-dependent but generalizes across PER sets within a model, and token-level and self-attention analyses provide mechanistic insights for designing privacy-aware prompts in NER systems.

Abstract

Training data memorization in language models impacts model capability (generalization) and safety (privacy risk). This paper focuses on analyzing prompts' impact on detecting the memorization of 6 masked language model-based named entity recognition models. Specifically, we employ a diverse set of 400 automatically generated prompts, and a pairwise dataset where each pair consists of one person's name from the training set and another name out of the set. A prompt completed with a person's name serves as input for getting the model's confidence in predicting this name. Finally, the prompt performance of detecting model memorization is quantified by the percentage of name pairs for which the model has higher confidence for the name from the training set. We show that the performance of different prompts varies by as much as 16 percentage points on the same model, and prompt engineering further increases the gap. Moreover, our experiments demonstrate that prompt performance is model-dependent but does generalize across different name sets. A comprehensive analysis indicates how prompt performance is influenced by prompt properties, contained tokens, and the model's self-attention weights on the prompt.

Figures (11)

• Figure 1: Memorization detection of a NER model using diverse prompts and the pairwise dataset. $P^{In}_*$ and $P^{Out}_*$ refer to a person's name in and out of the training data respectively.
• Figure 2: (a): Mean and standard deviation of M-MEM scores of all prompts across models on the dev set. (b): Correlations (Kendall's $\tau$) of the M-MEM scores of prompts across models. Left: correlations for the dev set scores. Middle: correlations for the test set scores. Right: correlations between dev and test set scores.
• Figure 3: M-MEM scores grouped by different prompt properties on the dev set.
• Figure 4: Analysis of the token importance in prompts and the M-MEM scores for the BERT-B model. The best-performing (left) and the worst-performing (right) prompts were selected on the dev set. The heatmaps are generated with leave-one-token-out: at each step, the least important token is removed from the best-performing prompt, and the most important token is removed from the worst-performing prompt. The normalized token importance scores are under the corresponding tokens. The removed tokens are underlined.
• Figure 5: Attention heatmaps of the best and the worst prompts of BERT-B on the dev set averaged over all attention heads and layers. The attention weights corresponding to each prompt's "MASK" token are used and averaged over the In-train and Out-train $PER$s separately.

Theorems & Definitions (2)

• Definition 1
• Definition 2