Bag of Tricks for Training Data Extraction from Language Models

TL;DR

The paper tackles privacy leakage in language models by studying training data extraction and proposing a bag-of-tricks to improve a generate-then-rank pipeline. It systematically evaluates generation-time tricks (sampling, distribution adjustments, exposure-bias mitigation, look-ahead) and ranking-time tricks (alternative scoring criteria) on GPT-Neo 1.3B, showing that several previously overlooked techniques yield substantial gains while interactions between tricks can be complex. A key contribution is establishing a stronger baseline for targeted data extraction and highlighting which tricks scale with model size or interact poorly, informing future privacy evaluation and defense work. The work also provides practical guidelines and an open-source implementation to encourage end-to-end exploration of compatible methods for data extraction and privacy risk assessment.

Abstract

With the advance of language models, privacy protection is receiving more attention. Training data extraction is therefore of great importance, as it can serve as a potential tool to assess privacy leakage. However, due to the difficulty of this task, most of the existing methods are proof-of-concept and still not effective enough. In this paper, we investigate and benchmark tricks for improving training data extraction using a publicly available dataset. Because most existing extraction methods use a pipeline of generating-then-ranking, i.e., generating text candidates as potential training data and then ranking them based on specific criteria, our research focuses on the tricks for both text generation (e.g., sampling strategy) and text ranking (e.g., token-level criteria). The experimental results show that several previously overlooked tricks can be crucial to the success of training data extraction. Based on the GPT-Neo 1.3B evaluation results, our proposed tricks outperform the baseline by a large margin in most cases, providing a much stronger baseline for future research. The code is available at https://github.com/weichen-yu/LM-Extraction.

Figures (9)

• Figure 1: Overview for the bag of tricks explored in this work, with an evaluation of precision ($\mathcal{M}_P$). Bars in pink denote the methods in the improved suffix generation, and bars in orange denote the methods in the improved suffix ranking. The dashed bars indicate the best method in each category.
• Figure 2: Histogram of token logits. The histogram depicts the distribution of logit values obtained from 1,000 suffixes, each containing 50 tokens. A spline interpolation technique is employed to smooth the histogram, with the original histogram included in the appendix for reference.
• Figure 3: Experimental results under different values of top-$k$, nucleus-$\eta$, typical-$\phi$ and temperature $T$. All results are reported on 5 trials. The y-axis left denotes precision (%)$\left(\uparrow\right)$, and right denotes Hamming distance $\left(\downarrow\right)$.
• Figure 4: Generated token length w.r.t. token precision (%) for the $n$-th generated token. The generated suffix length is 50.
• Figure 5: Histogram of the rank of the ground truth perplexity. The x-axis represents the rank of the ground truth perplexity within a list of 100 suffix perplexities.

Theorems & Definitions (1)

• Definition 1