Robustifying Safety-Aligned Large Language Models through Clean Data Curation

TL;DR

This work tackles safety misalignment in LLMs arising from training-time jailbreaking by introducing Ctrl, a perplexity-driven data-curation framework that revises a fraction of texts to lower perplexity while preserving readability and usefulness. Ctrl is designed to function without attacker-specific knowledge, strengthening safety during pre-training and downstream fine-tuning against both pre-training data tampering and post-training attacks, with beam-search guided revisions and open-ended generation. Empirical results show substantial reductions in harmful outputs and attack success rates—up to about 71% in certain 5% harm scenarios—alongside maintained or improved helpfulness across multiple models, illustrating the practicality of a data-centric defense. The approach highlights the importance of data quality and distribution in robust safety alignment and provides a scalable pathway to mitigate training-based jailbreaking while keeping LLMs useful and informative.

Abstract

Large language models (LLMs) are vulnerable when trained on datasets containing harmful content, which leads to potential jailbreaking attacks in two scenarios: the integration of harmful texts within crowdsourced data used for pre-training and direct tampering with LLMs through fine-tuning. In both scenarios, adversaries can compromise the safety alignment of LLMs, exacerbating malfunctions. Motivated by the need to mitigate these adversarial influences, our research aims to enhance safety alignment by either neutralizing the impact of malicious texts in pre-training datasets or increasing the difficulty of jailbreaking during downstream fine-tuning. In this paper, we propose a data curation framework designed to counter adversarial impacts in both scenarios. Our method operates under the assumption that we have no prior knowledge of attack details, focusing solely on curating clean texts. We introduce an iterative process aimed at revising texts to reduce their perplexity as perceived by LLMs, while simultaneously preserving their text quality. By pre-training or fine-tuning LLMs with curated clean texts, we observe a notable improvement in LLM robustness regarding safety alignment against harmful queries. For instance, when pre-training LLMs using a crowdsourced dataset containing 5\% harmful instances, adding an equivalent amount of curated texts significantly mitigates the likelihood of providing harmful responses in LLMs and reduces the attack success rate by 71\%. Our study represents a significant step towards mitigating the risks associated with training-based jailbreaking and fortifying the secure utilization of LLMs.

Figures (9)

• Figure 1: An illustration of two training-based attacks in Scenario I and II.
• Figure 2: Text perplexity on (a) safety-aligned and (b) jailbroken Llama-3-8B. We use security-sensitive queries from AdvBench and BeaverTails to construct our safety and harmfulness datasets, pairing them with safe and harmful responses, respectively. Additionally, we utilize Alpaca, Dolly, and a portion of BeaverTails (with queries irrelevant to security topics) as our general-domain datasets.
• Figure 3: An illustration of how Ctrl works. PPL:perplexity, Rd: readability, Hp: helpfulness.
• Figure 4: Perplexity variation with changes in temperature and top-p, measured on a randomly selected (query, response) pair using Llama-3-8B.
• Figure 5: We measure the changes in the following guideline metrics over 1, 3, and 5 iterations of beam search: (a) Perplexity, (b) Readability, and (c) Helpfulness. The "Org" values represent the original metrics before applying Ctrl.