Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence
Shaopeng Fu, Liang Ding, Jingfeng Zhang, Di Wang
TL;DR
The paper addresses jailbreaking safety for LLMs and the substantial compute cost of using long adversarial prompts during training. It theoretically establishes that defending against a suffix jailbreak of length Θ($M$) can be achieved by adversarial training with suffix length Θ($√{M}$), through an in-context learning framework with linear transformers and a robust generalization bound that scales with Θ($√{M_{test}}/M_{train}$). It then derives a closed-form surrogate AT solution and proves convergence under continuous gradient flow, linking AT dynamics to a specific constraint on the learned blocks. Empirically, the authors validate the theory by training on five open LLMs and evaluating against multiple suffix jailbreak attacks, showing that short-length AT substantially reduces attack success while maintaining useful utility, and that the observed correlations hold across attacks and models. The work suggests practical, compute-efficient defenses for long-length jailbreaks using short-length adversarial training, with broad implications for safer deployment of LLMs.
Abstract
Jailbreak attacks against large language models (LLMs) aim to induce harmful behaviors in LLMs through carefully crafted adversarial prompts. To mitigate attacks, one way is to perform adversarial training (AT)-based alignment, i.e., training LLMs on some of the most adversarial prompts to help them learn how to behave safely under attacks. During AT, the length of adversarial prompts plays a critical role in the robustness of aligned LLMs. While long-length adversarial prompts during AT might lead to strong LLM robustness, their synthesis however is very resource-consuming, which may limit the application of LLM AT. This paper focuses on adversarial suffix jailbreak attacks and unveils that to defend against a jailbreak attack with an adversarial suffix of length $Θ(M)$, it is enough to align LLMs on prompts with adversarial suffixes of length $Θ(\sqrt{M})$. Theoretically, we analyze the adversarial in-context learning of linear transformers on linear regression tasks and prove a robust generalization bound for trained transformers. The bound depends on the term $Θ(\sqrt{M_{\text{test}}}/M_{\text{train}})$, where $M_{\text{train}}$ and $M_{\text{test}}$ are the numbers of adversarially perturbed in-context samples during training and testing. Empirically, we conduct AT on popular open-source LLMs and evaluate their robustness against jailbreak attacks of different adversarial suffix lengths. Results confirm a positive correlation between the attack success rate and the ratio of the square root of the adversarial suffix length during jailbreaking to the length during AT. Our findings show that it is practical to defend against "long-length" jailbreak attacks via efficient "short-length" AT. The code is available at https://github.com/fshp971/adv-icl.