Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning

Yanbo Wang, Minzheng Wang, Jian Liang, Lu Wang, Yongcan Yu, Ran He

TL;DR

The Adaptive Safe Context Learning (ASCL) framework is proposed, empowering the model to autonomously decide when to consult safety rules and how to generate the ongoing reasoning, and Inverse Frequency Policy Optimization is introduced to rebalance advantage estimates.

Abstract

While reasoning models have achieved remarkable success in complex reasoning tasks, their increasing power necessitates stringent safety measures. For safety alignment, the core challenge lies in the inherent trade-off between safety and utility. However, prevailing alignment strategies typically construct CoT training data with explicit safety rules via context distillation. This approach inadvertently limits reasoning capabilities by creating a rigid association between rule memorization and refusal. To mitigate the safety-utility trade-off, we propose the Adaptive Safe Context Learning (ASCL) framework to improve the reasoning given proper context. ASCL formulates safety alignment as a multi-turn tool-use process, empowering the model to autonomously decide when to consult safety rules and how to generate the ongoing reasoning. Furthermore, to counteract the preference for rule consultation during RL, we introduce Inverse Frequency Policy Optimization (IFPO) to rebalance advantage estimates. By decoupling rule retrieval and subsequent reasoning, our method achieves higher overall performance compared to baselines.

Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning

TL;DR

The Adaptive Safe Context Learning (ASCL) framework is proposed, empowering the model to autonomously decide when to consult safety rules and how to generate the ongoing reasoning, and Inverse Frequency Policy Optimization is introduced to rebalance advantage estimates.

Abstract

While reasoning models have achieved remarkable success in complex reasoning tasks, their increasing power necessitates stringent safety measures. For safety alignment, the core challenge lies in the inherent trade-off between safety and utility. However, prevailing alignment strategies typically construct CoT training data with explicit safety rules via context distillation. This approach inadvertently limits reasoning capabilities by creating a rigid association between rule memorization and refusal. To mitigate the safety-utility trade-off, we propose the Adaptive Safe Context Learning (ASCL) framework to improve the reasoning given proper context. ASCL formulates safety alignment as a multi-turn tool-use process, empowering the model to autonomously decide when to consult safety rules and how to generate the ongoing reasoning. Furthermore, to counteract the preference for rule consultation during RL, we introduce Inverse Frequency Policy Optimization (IFPO) to rebalance advantage estimates. By decoupling rule retrieval and subsequent reasoning, our method achieves higher overall performance compared to baselines.
Paper Structure (34 sections, 5 equations, 8 figures, 4 tables)

This paper contains 34 sections, 5 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Safety Alignment of Reasoning Models
  4. Context Learning in Model Safety
  5. Relation between Model Context and Safety-Refusal Trade-off
  6. Adaptive Safe Context Learning
  7. Setup
  8. Model settings.
  9. Evaluation benchmarks and metrics.
  10. Main Findings
  11. Boosting ASCL with Post-training
  12. Training Data Curation for Post-training
  13. Reward Designing for Reinforcement Learning
  14. Inverse-Frequency Policy Optimization
  15. Experiments
  16. ...and 19 more sections

Figures (8)

  • Figure 1: The impact of input context on safety-refusal tradeoff.
  • Figure 2: An overview of the post-training of the ASCL framework. Firstly, we illustrate the format of training data in the top-left corner, where ordinary samples without rule retrieval are colored in light blue , and samples with the tool-call mechanism are colored in light yellow . In the BC phrase, data are distilled according to the ASCL-ZS setting, and then paraphrased by another expert model to form the structured data. Following RL with IFPO boosts the model performance in an on-policy manner.
  • Figure 3: A qualitative comparison between the STAR-1-mix and ASCL. The sample is from XsTest rottger2024xstest.
  • Figure 4: Ablations on the ASCL framework. IFPO achieved the most balanced performance on average.
  • Figure 5: The comparisons of reward designing ablations.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Definition 4.1
  • Definition 4.2
  • Definition 4.3