Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines

TL;DR

The paper tackles safety gaps in reasoning models facing covert adversarial jailbreak prompts by introducing SGASA, a two-stage framework that self-generates safety guidelines and augments prompts, then internalizes them through SFT and DPO with LoRA fine-tuning. Data Pre-synthesis and Alignment Fine-tuning enable adaptive, model-driven defense that balances safety with minimizing unnecessary refusals. Across two backbones and three datasets, SGASA substantially improves safety metrics, with DPO-based variants delivering the strongest gains and demonstrating cross-dataset generalization. The work highlights scalable, automated safety alignment that can adapt to evolving jailbreak strategies and informs future directions for robust, guideline-driven defense mechanisms.

Abstract

Reasoning models have demonstrated remarkable capabilities in complex reasoning tasks. However, ensuring their safety against adversarial jailbreak prompts remains a critical challenge. Due to the covert and deceptive nature of such prompts, they can often evade built-in safety mechanisms and lead to the generation of harmful content. This underscores the need for an adaptive safety alignment approach that enables models to autonomously reinforce their defenses in response to adversarial inputs. This paper introduces the Synthesized Guideline-based Adaptive Safety Alignment (SGASA) framework, which internalizes model-generated safety guidelines to strengthen models' ability to enhance robustness against harmful adversarial prompts while minimizing unnecessary refusals of benign requests. SGASA consists of two key stages: Data Pre-synthesis, which generates safety guidelines and augmented prompts; and Alignment Fine-tuning, which leverages Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO) to embed these guidelines into the model. Extensive experiments across multiple datasets demonstrate that SGASA significantly improves model safety, validating its adaptive and scalable effectiveness.

Figures (7)

• Figure 1: Two adversarial prompt examples from the MaliciousEducator dataset.
• Figure 2: An example of safety guidelines generated using the DeepSeek-R1-Distill-Llama-8B, based on the jailbreak examples presented in Figure \ref{['fig:jailbreak case']}.
• Figure 3: Illustration of our framework.the top part shows data pre-synthesis, and the bottom part shows alignment fine-tuning. We use $q$ to denote the prompt, $g_n$ for a randomly sampled generated guideline, $y$ for the model's response, and $y^+$, $y^-$ for the positive and negative examples in DPO fine-tuning, respectively.
• Figure 4: Top: Instruction template used for generating safety guidelines. Bottom: Inference prompt template after applying the safety guidelines.
• Figure 5: Top: Instruction template used for prompt augmentation. Bottom: Instruction template used for prompt safety classification.