Investigating Adversarial Trigger Transfer in Large Language Models

TL;DR

The paper investigates whether adversarial triggers transfer across 13 open LLMs, contrasting models aligned via Preference Optimization (APO) with those aligned by Fine-Tuning (AFT). Using Greedy Coordinate Gradient on AdvBench, it shows APO models exhibit strong robustness to trigger transfer, often delaying or preventing jailbreaking, while AFT models are highly susceptible and exhibit rapid, cross-model transfer to unseen unsafe instructions ($\Delta$ASR). The study further demonstrates that triggers optimized on AFT models generalize to new unsafe instructions across diverse domains, underscoring vulnerabilities not captured by surface safety metrics. The findings call for more comprehensive safety evaluations, including automatic red-teaming and reporting attack effectiveness across multiple runs and a broad set of models, to better understand and mitigate jailbreak risks in aligned LLMs.

Abstract

Recent work has developed optimization procedures to find token sequences, called adversarial triggers, which can elicit unsafe responses from aligned language models. These triggers are believed to be highly transferable, i.e., a trigger optimized on one model can jailbreak other models. In this paper, we concretely show that such adversarial triggers are not consistently transferable. We extensively investigate trigger transfer amongst 13 open models and observe poor and inconsistent transfer. Our experiments further reveal a significant difference in robustness to adversarial triggers between models Aligned by Preference Optimization (APO) and models Aligned by Fine-Tuning (AFT). We find that APO models are extremely hard to jailbreak even when the trigger is optimized directly on the model. On the other hand, while AFT models may appear safe on the surface, exhibiting refusals to a range of unsafe instructions, we show that they are highly susceptible to adversarial triggers. Lastly, we observe that most triggers optimized on AFT models also generalize to new unsafe instructions from five diverse domains, further emphasizing their vulnerability. Overall, our work highlights the need for more comprehensive safety evaluations for aligned language models.

Figures (15)

• Figure 1: $\Delta$ Attack Success Rates ($\Delta$ASRs) for GCG triggers optimized using the best ensembles from zou_universal_2023. We report the mean $\Delta$ASR over three independently optimized triggers for each ensemble and use Llama-Guard for evaluating whether triggers jailbreak models. The dots show the $\Delta$ASR for each trigger. We find that triggers do not transfer to models trained for harmfulness with RLHF or DPO (i.e., APO models; shaded in ).
• Figure 2: $\Delta$ Attack Success Rates ($\Delta$ASRs) for BEAST (\ref{['fig:main_advbench_seen_beam_all']}) and AutoDAN (\ref{['fig:main_advbench_seen_autodan_all']}) triggers optimized on Vicuna-7B and Vicuna-7B/13B. We report the mean $\Delta$ASR over three independently optimized triggers for each ensemble and use Llama-Guard for evaluating whether triggers jailbreak models. The dots show the $\Delta$ASR for each trigger. APO models are shaded in .
• Figure 3: A reproduction of Figure 3 from zou_universal_2023 using four models investigated in our study also present in the original work. We include results for Llama2-7B-Chat but note no Llama2-7B-Chat transfer results are reported in zou_universal_2023. We report the ASRs for triggers optimized on $25$ random AdvBench examples evaluated against $388$ held-out instructions. We use the string-based metric from zou_universal_2023 for evaluating whether triggers jailbreak models.
• Figure 4: $\Delta$ASRs for triggers optimized on APO models (source) transferred to different models. We report mean $\Delta$ASRs over three independently optimized triggers for each source ensemble. See \ref{['fig:safety_advbench_seen_all']} in \ref{['sec:app_additional_results']} for results for additional ensembles.
• Figure 5: Left (a): $\Delta$ASRs through $500$ optimization steps. We report the mean $\Delta$ASR at each step over three independently optimized triggers for the APO (shaded in ) and AFT models. Right (b): $\Delta$ASRs for triggers directly optimized on and triggers transferred to APO (shaded in ) and AFT models. For triggers directly optimized on models (top right), the dots show the $\Delta$ASR for each trigger. For triggers transferred to models (bottom right), the dots show the $\Delta$ASRs for the best five triggers.