Data to Defense: The Role of Curation in Customizing LLMs Against Jailbreaking Attacks

TL;DR

The paper addresses jailbreaking risks that arise during LLM customization by proposing Data-to-Defense (D2D), a data curation framework that adaptively revises any dataset to increase perplexity while embedding safety implications. D2D can be applied before, during, or after customization, and its curated data is integrated into the fine-tuning process without adding inference overhead. Key contributions include seed-set preparation of safety keywords, beam-search guided text revision, and a validation showing up to 100% safe responses across multiple models while preserving general-domain usefulness. The findings indicate that D2D can robustly mitigate jailbreaking through a data-centric approach, with ablation results underscoring the importance of seed data and sampling, and perplexity analyses showing safer knowledge transfer without compromising task performance.

Abstract

Large language models (LLMs) are widely adapted for downstream applications through fine-tuning, a process named customization. However, recent studies have identified a vulnerability during this process, where malicious samples can compromise the robustness of LLMs and amplify harmful behaviors-an attack commonly referred to as jailbreaking. To address this challenge, we propose an adaptive data curation approach allowing any text to be curated to enhance its effectiveness in counteracting harmful samples during customization. To avoid the need for additional defensive modules, we further introduce a comprehensive mitigation framework spanning the lifecycle of the customization process: before customization to immunize LLMs against future jailbreak attempts, during customization to neutralize risks, and after customization to restore compromised models. Experimental results demonstrate a significant reduction in jailbreaking effects, achieving up to a 100% success rate in generating safe responses. By combining adaptive data curation with lifecycle-based mitigation strategies, this work represents a solid step forward in mitigating jailbreaking risks and ensuring the secure adaptation of LLMs.

Figures (7)

• Figure 1: An illustration of (a) Jailbreaking attack through fine-tuning (b)-(d) our proposed curation-based defense by including data in different stages of customization workflow.
• Figure 2: An illustration of how D2D works, where ➀➁➂ represent generated texts through output sampling. In this case, ➀ has lower perplexity, while ➁ demonstrates poor helpfulness. As a result, the beam search selects ➂ for the next round of output sampling. Perplexity is measured by an LLM that needs to be robustified, and helpfulness is rated by GPT-4o using prompts in Appendix \ref{['ap:helpfulness']}.
• Figure 3: Change in perplexity (y-axis) between (a) a jailbroken and (b) a mitigated Llama-3-8B, evaluated using safe answers from $\mathcal{D}_\texttt{security}$, original $\mathcal{D}_\texttt{general}$, and harmful answers from $\mathcal{D}_\texttt{security}$ (left-to-right boxes).
• Figure 4: Safety rate (SR) of LLM responses with varying volumes of curated and harmful texts. The volume is measured by their ratios within the fine-tuning dataset. More results are shown in Figure \ref{['fig:factor_other']}.
• Figure 5: SR of varying beam-search iterations.

Theorems & Definitions (1)

• Definition 1: Safety Implication