Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models

TL;DR

The paper addresses the persistent risk that jailbreaking can undermine safety in large language models and hypothesizes that adversarial suffixes are not mere bugs but carry transferable features. It introduces a universal suffix extractor and a PCC-based analysis to quantify suffix influence versus prompts, and validates three experiments showing that benign features can be turned into transferable adversarial suffixes, jailbreaking suffixes can contain meaningful features, and benign data used in fine-tuning can erode safety alignment. The findings demonstrate that safety guarantees can be compromised even with benign data, especially when dominant suffix features are learned or reinforced during fine-tuning, highlighting the need for robust alignment techniques and careful data handling in model customization. The work suggests that defense against jailbreaking must consider latent benign features and that safe fine-tuning strategies are critical for maintaining safety in deployed LLMs.

Abstract

Despite significant ongoing efforts in safety alignment, large language models (LLMs) such as GPT-4 and LLaMA 3 remain vulnerable to jailbreak attacks that can induce harmful behaviors, including through the use of adversarial suffixes. Building on prior research, we hypothesize that these adversarial suffixes are not mere bugs but may represent features that can dominate the LLM's behavior. To evaluate this hypothesis, we conduct several experiments. First, we demonstrate that benign features can be effectively made to function as adversarial suffixes, i.e., we develop a feature extraction method to extract sample-agnostic features from benign dataset in the form of suffixes and show that these suffixes may effectively compromise safety alignment. Second, we show that adversarial suffixes generated from jailbreak attacks may contain meaningful features, i.e., appending the same suffix to different prompts results in responses exhibiting specific characteristics. Third, we show that such benign-yet-safety-compromising features can be easily introduced through fine-tuning using only benign datasets. As a result, we are able to completely eliminate GPT's safety alignment in a blackbox setting through finetuning with only benign data. Our code and data is available at \url{https://github.com/suffix-maybe-feature/adver-suffix-maybe-features}.

Figures (31)

• Figure 1: An example showing how a suffix generated from a benign dataset to capture strong transferable benign feature alters model responses to both benign and harmful prompts, i.e., appending the suffix to a benign prompt causes the model to produce a response with a structured format; appending it to a harmful prompt induces a harmful response with a similar format. This example shows that strong benign features may compromise safety alignment.
• Figure 2: Overview of the universal feature extractor that optimizes a suffix (soft prompt) to capture sample-agnostic features from target datasets. For example, when applied to format-specific benign dataset, the generated suffix causes model responses to consistently follow that format style when appended to inputs.
• Figure 7: Poem Response System Template
• Figure 8: Structure Response System Template
• Figure 9: Basic Transferability Evaluation