Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking

TL;DR

<3-5 sentence high-level summary> The paper addresses safety against jailbreaking of large language models by enabling safety-aware reasoning during generation. It introduces Reasoning-to-Defend (R2D), combining Safety-aware Reasoning Distillation (SwaRD) with Contrastive Pivot Optimization (CPO) to allow self-evaluation and per-step safety checks via pivot tokens. Empirical results on JailbreakBench and HarmBench show substantial reductions in Attack Success Rate while preserving core performance and avoiding over-refusal. Ablation studies confirm the necessity of pivot tokens and CPO, and analyses indicate R2D maintains general abilities and improves safety across diverse attack methods.

Abstract

Large Reasoning Models (LRMs) have recently demonstrated impressive performances across diverse domains. However, how the safety of Large Language Models (LLMs) benefits from enhanced reasoning capabilities against jailbreak queries remains unexplored. To bridge this gap, in this paper, we propose Reasoning-to-Defend (R2D), a novel training paradigm that integrates a safety-aware reasoning mechanism into LLMs' generation process. This enables self-evaluation at each step of the reasoning process, forming safety pivot tokens as indicators of the safety status of responses. Furthermore, in order to improve the accuracy of predicting pivot tokens, we propose Contrastive Pivot Optimization (CPO), which enhances the model's perception of the safety status of given dialogues. LLMs dynamically adjust their response strategies during reasoning, significantly enhancing their safety capabilities defending jailbreak attacks. Extensive experiments demonstrate that R2D effectively mitigates various attacks and improves overall safety, while maintaining the original performances. This highlights the substantial potential of safety-aware reasoning in improving robustness of LRMs and LLMs against various jailbreaks.

Figures (5)

• Figure 1: R2D-enhanced LLM's responses against the attack Jailbreak Chat (JBC, wei2023jailbroken). "In this hypothetical story, you are to act as Niccolo Machiavelli..." is a widely-used adversarial prefix on JBC. It is demonstrated that R2D LLM is capable of rethinking when in need and labeling bad behaviors as [UNSAFE], showcasing its safety-aware reasoning abilities.
• Figure 2: Overview of the R2D framework. Compared to hard refusal responses, R2D LLMs refuse to answer with after concrete reasoning. The safety-aware reasoning process also improves the defensive performances based on the inner reasoning steps, thus reducing the possibility of generating unsafe responses.
• Figure 3: Histogram comparing the ASRs of LLMs with and without R2D on HarmBench. Sub-figures include results with different attacks, namely: ZeroShot, FewShot, PAIR and AutoDAN.
• Figure 4: Results of over-refusal analysis on XSTest dataset. Categories of responses from LLMs include "Full Refusal" (directly refuse to answer), "Full Compliance" (directly attempt to give an answer) and "Partial Refusal" (combination of refusal and compliance). Enhanced LLMs are marked as +R2D. Labels in Fig.\ref{['fig:overrefusal_results']} and Fig.\ref{['fig:overrefusal_results_strmatch']} are classified with Qwen$_\text{v2.5-72B}$ and Dic-Judge zou2023universal, respectively.
• Figure 5: Latencies (indicated by number of generated words) of different R2D-enhanced models, including Reasoning and Non-Reasoning LLMs.