Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training

TL;DR

The paper identifies a refusal position bias in safety-tuning data that impairs LLMs' ability to refuse unsafe content. It introduces Decoupled Refusal Training (DeRTa), combining MLE with Harmful Response Prefix and Reinforced Transition Optimization to enable safety refusals at any position in a response. Across LLaMA3 and Mistral families and six attack scenarios, DeRTa substantially improves safety with minimal impact on helpfulness, outperforming strong baselines and even challenging completion-type attacks. The approach shows robust performance across model sizes and languages, though it acknowledges limitations such as remaining adaptive threats and the need for broader jailbreak coverage.

Abstract

This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) by identifying and tackling a refusal position bias within safety tuning data, which compromises the models' ability to appropriately refuse generating unsafe content. We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position, significantly enhancing their safety capabilities. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model families across six attack scenarios, demonstrates that our method not only improves model safety without compromising performance but also surpasses baseline methods in defending against attacks.

Figures (14)

• Figure 1: Illustration of (a) the standard safety tuning, (b) ours method, (c) MLE with Harmful Prefix, and (d) RTO. In our method, we teach the model to recognize and halt the generation of unsafe content when they detect potential risks. The transition from harmful response to safety refusal only occurs once in MLE with Harmful Prefix (the dashed square), while in RTO we simulate the transition at every position within the full harmful response sequence.
• Figure 2: LLMs using our approach can refuse to answer whenever they feel it is unsafe, even if they are already at a later position in the response.
• Figure 3: The ASR of six attacks on our approach and the baselines. This experiment is conducted on LLaMA3-70B.
• Figure 4: Position distribution of where the refuse token, like "sorry", appears for safe responses.
• Figure 5: Comparison to DPO with the same safety data.