Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rules

Di Wu, Yanyan Zhao, Xin Lu, Mingzhe Li, Bing Qin

TL;DR

STAR-S addresses safety alignment against jailbreak attacks by enabling models to reason about safety rules before responding, using a self-taught loop of reasoning, reflection, and supervised fine-tuning. The method combines a Rules Design with a flawed reasoning prefix to provoke self-reflection, plus Reflection Enhancement via safety hints, and a rejection-sampling based fine-tuning loop. Empirical results across six jailbreak benchmarks and two over-refusal benchmarks show STAR-S improves jailbreak safety while maintaining general capabilities, and balances safety with over-refusal better than baselines. The work also investigates data distillation from other models, safety in agent scenarios, and discusses limitations and future directions.

Abstract

Defending against jailbreak attacks is crucial for the safe deployment of Large Language Models (LLMs). Recent research has attempted to improve safety by training models to reason over safety rules before responding. However, a key issue lies in determining what form of safety reasoning effectively defends against jailbreak attacks, which is difficult to explicitly design or directly obtain. To address this, we propose \textbf{STAR-S} (\textbf{S}elf-\textbf{TA}ught \textbf{R}easoning based on \textbf{S}afety rules), a framework that integrates the learning of safety rule reasoning into a self-taught loop. The core of STAR-S involves eliciting reasoning and reflection guided by safety rules, then leveraging fine-tuning to enhance safety reasoning. Repeating this process creates a synergistic cycle. Improvements in the model's reasoning and interpretation of safety rules allow it to produce better reasoning data under safety rule prompts, which is then utilized for further training. Experiments show that STAR-S effectively defends against jailbreak attacks, outperforming baselines. Code is available at: https://github.com/pikepokenew/STAR_S.git.

STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rules

TL;DR

STAR-S addresses safety alignment against jailbreak attacks by enabling models to reason about safety rules before responding, using a self-taught loop of reasoning, reflection, and supervised fine-tuning. The method combines a Rules Design with a flawed reasoning prefix to provoke self-reflection, plus Reflection Enhancement via safety hints, and a rejection-sampling based fine-tuning loop. Empirical results across six jailbreak benchmarks and two over-refusal benchmarks show STAR-S improves jailbreak safety while maintaining general capabilities, and balances safety with over-refusal better than baselines. The work also investigates data distillation from other models, safety in agent scenarios, and discusses limitations and future directions.

Abstract

Defending against jailbreak attacks is crucial for the safe deployment of Large Language Models (LLMs). Recent research has attempted to improve safety by training models to reason over safety rules before responding. However, a key issue lies in determining what form of safety reasoning effectively defends against jailbreak attacks, which is difficult to explicitly design or directly obtain. To address this, we propose \textbf{STAR-S} (\textbf{S}elf-\textbf{TA}ught \textbf{R}easoning based on \textbf{S}afety rules), a framework that integrates the learning of safety rule reasoning into a self-taught loop. The core of STAR-S involves eliciting reasoning and reflection guided by safety rules, then leveraging fine-tuning to enhance safety reasoning. Repeating this process creates a synergistic cycle. Improvements in the model's reasoning and interpretation of safety rules allow it to produce better reasoning data under safety rule prompts, which is then utilized for further training. Experiments show that STAR-S effectively defends against jailbreak attacks, outperforming baselines. Code is available at: https://github.com/pikepokenew/STAR_S.git.
Paper Structure (44 sections, 3 equations, 4 figures, 6 tables)

This paper contains 44 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLMs Safety
  4. Large Reasoning Models
  5. Self-Improvement
  6. Methodology
  7. Reasoning Generation
  8. Rules Design
  9. Flawed Reasoning Prefix
  10. Reflection Enhancement
  11. Supervised Fine-Tuning
  12. Experimental Setup
  13. Models
  14. Dataset
  15. Evaluation Setup
  16. ...and 29 more sections

Figures (4)

  • Figure 1: Overview of the self-taught reasoning process. STAR-S guides the model to reason and reflect on safety rules; the model is then fine-tuned on this data, repeating this process to improve safety rule reasoning.
  • Figure 2: Overview of the STAR-S method. In the reasoning generation stage, the model generates reasoning data conditioned on flawed prefixes. During reflection enhancement, additional hints are provided to guide the model's self-reflection. In the supervised fine-tuning stage, the model is trained to apply safety rules during reasoning. This model then serves as the reasoner for data generation in subsequent iterations.
  • Figure 3: The trade-off between safety performance and over-refusal rate for different methods. STAR-S achieves a superior balance between these two metrics.
  • Figure 4: Relationship between jailbreak safety score and iteration rounds under different ablation settings. Our proposed STAR-S method consistently improves its safety score with increasing iterations. A similar upward trend is observed across all ablation conditions.