Extracting alignment data in open models

TL;DR

The paper reveals that significant alignment data can be extracted from open-weight LLMs by exploiting post-training chat templates and using embedding-based semantic memorisation to detect leakage. It demonstrates that embedding metrics uncover far more memorisation than traditional string matching, potentially by at least $10\times$, and shows that synthetic data derived from extracted outputs can retrain a base model to recover substantial portions of the original performance, exposing distillation as a leakage channel. The work extends to large-scale SFT data and RL data, showing consistent leakage across modalities and highlighting practical risks for privacy and competitive advantage. While the attack is demonstrated on open models, it raises important questions about mitigation and governance of training data used in alignment pipelines.

Abstract

In this work, we show that it is possible to extract significant amounts of alignment training data from a post-trained model -- useful to steer the model to improve certain capabilities such as long-context reasoning, safety, instruction following, and maths. While the majority of related work on memorisation has focused on measuring success of training data extraction through string matching, we argue that embedding models are better suited for our specific goals. Distances measured through a high quality embedding model can identify semantic similarities between strings that a different metric such as edit distance will struggle to capture. In fact, in our investigation, approximate string matching would have severely undercounted (by a conservative estimate of $10\times$) the amount of data that can be extracted due to trivial artifacts that deflate the metric. Interestingly, we find that models readily regurgitate training data that was used in post-training phases such as SFT or RL. We show that this data can be then used to train a base model, recovering a meaningful amount of the original performance. We believe our work exposes a possibly overlooked risk towards extracting alignment data. Finally, our work opens up an interesting discussion on the downstream effects of distillation practices: since models seem to be regurgitating aspects of their training set, distillation can therefore be thought of as indirectly training on the model's original dataset.

Figures (20)

• Figure 1: An overview of the data extraction process. A sample generated using the prompt <|endoftext|><|user|>\\ n is used as a query for a vector search to find the best semantic match from the training data. The figure highlights the difference between a high embedding similarity score (0.97) and a much lower string-match similarity score (0.68), demonstrating that semantic similarity is more effective for detecting this form of data memorisation.
• Figure 2: Generations from Qwen2.5 7B Instruct using <|im_start|>user\\ n as a prefill. Qualitatively, the generations resemble potential post-training samples.
• Figure 3: (Left) Generation from OLMo 2 13B. (Right) True post-training sample. Neural embeddings match provide a score of $0.986$ using gemini-embedding-001lee2025gemini, while normalised Levenshtein similarity provides a match score of $0.699$, heavily penalising differences in the options, even though the semantics remain identical. When computing similarities we always strip out special tokens, highlighted in blue. We report in the Appendix (Section \ref{['app:string-matching-bad-cases']}) more of such examples.
• Figure 4: Histograms illustrating the distribution of memorisation scores, measured by Levenshtein and Indel similarity scores. For each generation, we find the largest similarity score in the post-training set of OLMo 2.
• Figure 5: Histograms illustrating the distribution of embedding scores generated with gemini-embedding-001 (left) and scatter plot comparing the embedding scores to the Levenshtein distance, with points coloured by string length (right).