Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning

Kehua Feng, Keyan Ding, Yuhao Wang, Menghan Li, Fanjunduo Wei, Xinda Wang, Qiang Zhang, Huajun Chen

TL;DR

SAFER tackles safety gaps in large language models by enforcing structured Ex-Ante reasoning prior to response generation. It combines supervised fine-tuning with reasoning traces and step-level Ex-Ante Reasoning Preference Optimization (ERPO) to ground safety judgments in explicit rules while balancing helpfulness and efficiency. The method demonstrates improved safety robustness on open-source LLMs across diverse benchmarks, including scientific safety tasks, without sacrificing general performance. The work advances transparent, rule-grounded safety alignment and discusses latency trade-offs, outlining directions for more adaptive and scalable safety reasoning.

Abstract

Recent advancements in large language models (LLMs) have accelerated progress toward artificial general intelligence, yet their potential to generate harmful content poses critical safety challenges. Existing alignment methods often struggle to cover diverse safety scenarios and remain vulnerable to adversarial attacks. In this work, we propose SAFER, a framework for Safety Alignment via eFficient Ex-Ante Reasoning. Our approach instantiates structured Ex-Ante reasoning through initial assessment, rule verification, and path calibration, and embeds predefined safety rules to provide transparent and verifiable safety judgments. Specifically, our approach consists of two training stages: (1) supervised fine-tuning with synthetic traces to teach the multi-stage Ex-Ante reasoning, and (2) step-level reasoning preference optimization to jointly enhance safety, utility, and efficiency. Experiments on multiple open-source LLMs demonstrate that SAFER significantly enhances safety performance while maintaining helpfulness and response efficiency.

SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning

TL;DR

SAFER tackles safety gaps in large language models by enforcing structured Ex-Ante reasoning prior to response generation. It combines supervised fine-tuning with reasoning traces and step-level Ex-Ante Reasoning Preference Optimization (ERPO) to ground safety judgments in explicit rules while balancing helpfulness and efficiency. The method demonstrates improved safety robustness on open-source LLMs across diverse benchmarks, including scientific safety tasks, without sacrificing general performance. The work advances transparent, rule-grounded safety alignment and discusses latency trade-offs, outlining directions for more adaptive and scalable safety reasoning.

Abstract

Recent advancements in large language models (LLMs) have accelerated progress toward artificial general intelligence, yet their potential to generate harmful content poses critical safety challenges. Existing alignment methods often struggle to cover diverse safety scenarios and remain vulnerable to adversarial attacks. In this work, we propose SAFER, a framework for Safety Alignment via eFficient Ex-Ante Reasoning. Our approach instantiates structured Ex-Ante reasoning through initial assessment, rule verification, and path calibration, and embeds predefined safety rules to provide transparent and verifiable safety judgments. Specifically, our approach consists of two training stages: (1) supervised fine-tuning with synthetic traces to teach the multi-stage Ex-Ante reasoning, and (2) step-level reasoning preference optimization to jointly enhance safety, utility, and efficiency. Experiments on multiple open-source LLMs demonstrate that SAFER significantly enhances safety performance while maintaining helpfulness and response efficiency.

Paper Structure

This paper contains 41 sections, 7 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Safety Alignment Approaches
  4. Safety Evaluation and Red-Teaming
  5. Method
  6. Learning to Ex-Ante Reason via SFT
  7. I. Safety Rule Curation and Annotation
  8. II. Ex-Ante Reasoning Trace Generation
  9. Supervised Fine-Tuning
  10. Enhancing Ex-Ante Reasoning via ERPO
  11. 1) Safety: A more accurate reasoning trace is preferred.
  12. 2) Helpfulness: A more helpful response is desired.
  13. 3) Length: Reasoning process should be adaptive.
  14. Weighted DPO Objective.
  15. Experiments
  16. ...and 26 more sections

Figures (5)

  • Figure 1: Existing methods can prevent queries with obvious risks, but there are still "edge" cases that cannot be covered. For example, replacing "sarin gas" with its SMILES notation may bypass detection by the model.
  • Figure 2: Illustration of the proposed SAFER framework, which comprises the following two stages: (1) In the SFT stage, safety-tuning data incorporating Ex-Ante reasoning trace are constructed to train the model to generate Ex-Ante reasoning before responding. (2) In the ERPO stage, preference pairs are built to refine safety judgment, response helpfulness, and reasoning conciseness.
  • Figure 3: Illustration of generating preference data for ERPO. We separately synthesize preferences for unsafe and safe prompts based on three-dimensional safety principles.
  • Figure 4: Changes in Best-of-N ASR (left) and Worst-of-N ASR (right) on HarmBench with test-time scaling.
  • Figure 5: Safety benchmark performance (ASR $\downarrow$) with or without safety rules during SFT.