Efficient Safety Retrofitting Against Jailbreaking for LLMs

TL;DR

This work examines the effectiveness of Direct Preference Optimization (DPO) for safety alignment of LLMs against jailbreaking while prioritizing data efficiency. By constructing and releasing the Egida dataset (with 61,830 unsafe instances across 27 topics and 18 jailbreaking styles) and extending it with safe responses and human annotations, the authors train safety-aligned models across Llama-3.1 and Qwen families. They demonstrate a 10–30% reduction in Attack Success Rate on unseen jailbreaking styles with as few as ~2,000 training samples at low cost, while also revealing limitations such as over-refusal and variability across model families. The study provides practical guidance on data composition, model selection, and the trade-offs in safety vs. general performance, and releases all resources to foster reproducibility and further research.

Abstract

Direct Preference Optimization (DPO) is an efficient alignment technique that steers LLMs towards preferable outputs by training on preference data, bypassing the need for explicit reward models. Its simplicity enables easy adaptation to various domains and safety requirements. This paper examines DPO's effectiveness in model safety against jailbreaking attacks while minimizing data requirements and training costs. We introduce Egida, a dataset expanded from multiple sources, which includes 27 different safety topics and 18 different attack styles, complemented with synthetic and human labels. This data is used to boost the safety of state-of-the-art LLMs (Llama-3.1-8B/70B-Instruct, Qwen-2.5-7B/72B-Instruct) across topics and attack styles. In addition to safety evaluations, we assess their post-alignment performance degradation in general purpose tasks, and their tendency to over refusal. Following the proposed methodology, trained models reduce their Attack Success Rate by 10%-30%, using small training efforts (2,000 samples) with low computational cost (3\$ for 8B models, 20\$ for 72B models). Safety aligned models generalize to unseen topics and attack styles, with the most successful attack style reaching a success rate around 5%. Size and family are found to strongly influence model malleability towards safety, pointing at the importance of pre-training choices. To validate our findings, a large independent assessment of human preference agreement with Llama-Guard-3-8B is conducted by the authors and the associated dataset Egida-HSafe is released. Overall, this study illustrates how affordable and accessible it is to enhance LLM safety using DPO while outlining its current limitations. All datasets and models are released to enable reproducibility and further research.

Figures (19)

• Figure 1: Topic frequency in the Egida dataset.
• Figure 2: Percentage of agreement between each human evaluator and Llama-Guard per topic. The first three bars in each column represent men (depicted in different shades of blue), while the last two bars represent women (depicted in salmon).
• Figure 3: Performance of the four models under study on the four evaluation safety benchmarks. Y axis shows performance in attack success rate (ASR, lower better), and X axis shows an increasing amount of data used for alignment. 'x' correspond to original model performance.
• Figure 4: For Llama-3.1-8B-Instruct, ASR (y axis) change for each attack style (left) and safety topic (right) in the Egida test set, while using an increasing amount of data (x axis) for DPO model alignment. Lower is better.
• Figure 5: Attack success rate (y axis, lower better) on the two most challenging datasets after models are aligned with an increasing number of attack styles (top two rows) or an increasing number of dangerous topics (bottom two rows).