Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Models

TL;DR

This work addresses the risk that large language models encode and leak training data when prompted with specially crafted sequences. It introduces the Special Characters Attack (SCA), which uses two symbol sets plus English letters to trigger memorized raw data outputs, and extends it with SCA-LB, which biases certain logits to amplify leakage. Across open-source and commercial LLMs, SCA extracts diverse content (emails, code, prompt templates, and chat messages) and can reveal information about the training corpus distribution; it also shows an energy-latency dimension through non-stopping verbose outputs. The findings underscore a fundamental vulnerability in LLM training data memorization and distribution, highlighting the need for defenses at dataset curation, tokenization, and alignment stages to improve robustness and privacy.

Abstract

Large language models (LLMs) have achieved remarkable performance on a wide range of tasks. However, recent studies have shown that LLMs can memorize training data and simple repeated tokens can trick the model to leak the data. In this paper, we take a step further and show that certain special characters or their combinations with English letters are stronger memory triggers, leading to more severe data leakage. The intuition is that, since LLMs are trained with massive data that contains a substantial amount of special characters (e.g. structural symbols {, } of JSON files, and @, # in emails and online posts), the model may memorize the co-occurrence between these special characters and the raw texts. This motivates us to propose a simple but effective Special Characters Attack (SCA) to induce training data leakage. Our experiments verify the high effectiveness of SCA against state-of-the-art LLMs: they can leak diverse training data, such as code corpus, web pages, and personally identifiable information, and sometimes generate non-stop outputs as a byproduct. We further show that the composition of the training data corpus can be revealed by inspecting the leaked data -- one crucial piece of information for pre-training high-performance LLMs. Our work can help understand the sensitivity of LLMs to special characters and identify potential areas for improvement.

Figures (5)

• Figure 1: Our proposed Special Characters Attack (SCA) successfully extracts email addresses, iCloud accounts, preset prompts (mostly likely from Midjourney), and GitHub links from LLMs.
• Figure 1: The five random combination methods of our proposed SCA, which can be grouped into in-set combination and cross-set combination methods with three sets.
• Figure 2: The next token probability distributions for SCA, random, and article/code sequences.
• Figure 2: Output length distribution of LLMs under SCA. The x-axis represents the length of the output as a percentage of the maximum token length. The y-axis represents the number of examples or instances that correspond to each output length percentage. Gemini and ChatGPT are more likely to generate longer responses. Llama-2-Chat and Falcon are more likely to generate longer outputs. ChatGLM is less likely to generate such verbose outputs.
• Figure 3: Content and language distributions of the leaked semantic outputs from LLMs.