Retracing the Past: LLMs Emit Training Data When They Get Lost
Myeongseob Ko, Nikhil Reddy Billa, Adam Nguyen, Charles Fleming, Ming Jin, Ruoxi Jia
TL;DR
The paper tackles the memorization leakage of training data in LLMs by introducing Confusion-Inducing Attacks (CIA), which systematically maximize token-level prediction entropy to drive models into high-uncertainty states that precede memorized data emission. It augments CIA with mismatched Supervised Fine-Tuning to weaken alignment for aligned models, enabling improved extraction rates without access to training data. Across unaligned models (e.g., Llama 1/2) and aligned models (e.g., Llama 2-Chat, Llama 3-Instruct variants), CIA and CIA+SFT outperform prior baselines, achieving verbatim matches up to $VM@50$ ~22% on unaligned models and up to ~6% on aligned models, with near-verbatim success around 18% under relaxed tolerance. The work also provides a practical verification pipeline via InfiniGram and an ablation study showing the role of entropy objectives and SFT mismatches in driving leakage. Overall, the findings establish a more systematic framework for assessing memorization risks and highlight a concrete physiological signal—the entropy spike—as a precursor to data regurgitation, with implications for privacy and copyright protections in LLM deployment.
Abstract
The memorization of training data in large language models (LLMs) poses significant privacy and copyright concerns. Existing data extraction methods, particularly heuristic-based divergence attacks, often exhibit limited success and offer limited insight into the fundamental drivers of memorization leakage. This paper introduces Confusion-Inducing Attacks (CIA), a principled framework for extracting memorized data by systematically maximizing model uncertainty. We empirically demonstrate that the emission of memorized text during divergence is preceded by a sustained spike in token-level prediction entropy. CIA leverages this insight by optimizing input snippets to deliberately induce this consecutive high-entropy state. For aligned LLMs, we further propose Mismatched Supervised Fine-tuning (SFT) to simultaneously weaken their alignment and induce targeted confusion, thereby increasing susceptibility to our attacks. Experiments on various unaligned and aligned LLMs demonstrate that our proposed attacks outperform existing baselines in extracting verbatim and near-verbatim training data without requiring prior knowledge of the training data. Our findings highlight persistent memorization risks across various LLMs and offer a more systematic method for assessing these vulnerabilities.