Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety

Geon-Hyeong Kim, Youngsoo Jang, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, Moontae Lee

TL;DR

SafeDPO addresses safety alignment in LLMs by directly optimizing a safety-constrained objective within a single policy-learning stage. It achieves this by reordering preferences with safety indicators and applying a DPO-style update to a transformed dataset, thereby eliminating the need to train explicit reward or cost models. The authors prove theoretical equivalence to the original safety objective, preserve optimality with a nonnegative offset parameter $\Delta$, and demonstrate competitive safety and helpfulness on PKU-SafeRLHF-30K with favorable data and memory efficiency. The approach offers a simple, scalable alternative to SafeRLHF and other multi-stage methods, with potential for online and multi-objective extensions and broader adoption in safety-aligned fine-tuning.

Abstract

As Large Language Models (LLMs) continue to advance and find applications across a growing number of fields, ensuring the safety of LLMs has become increasingly critical. To address safety concerns, recent studies have proposed integrating safety constraints into Reinforcement Learning from Human Feedback (RLHF). However, these approaches tend to be complex, as they encompass complicated procedures in RLHF along with additional steps required by the safety constraints. Inspired by Direct Preference Optimization (DPO), we introduce a new algorithm called SafeDPO, which is designed to directly optimize the safety alignment objective in a single stage of policy learning, without requiring relaxation. SafeDPO introduces only one additional hyperparameter to further enhance safety and requires only minor modifications to standard DPO. As a result, it eliminates the need to fit separate reward and cost models or to sample from the language model during fine-tuning, while still enhancing the safety of LLMs. Finally, we demonstrate that SafeDPO achieves competitive performance compared to state-of-the-art safety alignment algorithms, both in terms of aligning with human preferences and improving safety.

SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety

TL;DR

SafeDPO addresses safety alignment in LLMs by directly optimizing a safety-constrained objective within a single policy-learning stage. It achieves this by reordering preferences with safety indicators and applying a DPO-style update to a transformed dataset, thereby eliminating the need to train explicit reward or cost models. The authors prove theoretical equivalence to the original safety objective, preserve optimality with a nonnegative offset parameter , and demonstrate competitive safety and helpfulness on PKU-SafeRLHF-30K with favorable data and memory efficiency. The approach offers a simple, scalable alternative to SafeRLHF and other multi-stage methods, with potential for online and multi-objective extensions and broader adoption in safety-aligned fine-tuning.

Abstract

As Large Language Models (LLMs) continue to advance and find applications across a growing number of fields, ensuring the safety of LLMs has become increasingly critical. To address safety concerns, recent studies have proposed integrating safety constraints into Reinforcement Learning from Human Feedback (RLHF). However, these approaches tend to be complex, as they encompass complicated procedures in RLHF along with additional steps required by the safety constraints. Inspired by Direct Preference Optimization (DPO), we introduce a new algorithm called SafeDPO, which is designed to directly optimize the safety alignment objective in a single stage of policy learning, without requiring relaxation. SafeDPO introduces only one additional hyperparameter to further enhance safety and requires only minor modifications to standard DPO. As a result, it eliminates the need to fit separate reward and cost models or to sample from the language model during fine-tuning, while still enhancing the safety of LLMs. Finally, we demonstrate that SafeDPO achieves competitive performance compared to state-of-the-art safety alignment algorithms, both in terms of aligning with human preferences and improving safety.

Paper Structure

This paper contains 54 sections, 7 theorems, 33 equations, 4 figures, 25 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reinforcement Learning from Human Feedback
  4. Direct Alignment Algorithms
  5. Safety Alignment
  6. Related Works
  7. Preference Alignment
  8. Safety Alignment
  9. Direct Preference Optimization with Enhanced Safety
  10. Problem Formulation
  11. Deriving the SafeDPO Objective
  12. Enhancing the Safety of SafeDPO
  13. Theoretical Analysis for SafeDPO
  14. Experiments
  15. Experimental Setups
  16. ...and 39 more sections

Key Result

Proposition 4.0

Under Assumption ass:at_least_one_safe, the set of optimal solutions to Equation eq:modified_saferlhf_obj is equivalent to that of Equation eq:safety_alignment.

Figures (4)

  • Figure 1: Safe RLHF (left) and SafeDPO (right). The blue items indicate components additionally used in both SafeDPO and Safe RLHF compared to DPO, while the red items represent components additionally used in Safe RLHF compared to SafeDPO.
  • Figure 2: Helpfulness, Harmlessness and Harmless Ratio Evaluation. The Dashed line indicates the borderline between the safe and unsafe. In (a), the harmless ratio is represented by the proportion of cases where the cost is less than or equal to zero, and harmlessness is measured by the average negative cost value. In (b), the harmless ratio is defined as the proportion of cases where the cost is higher than five, and harmlessness is assessed by the average score on a scale from 0 to 10.
  • Figure 3: Harmlessness and Helpfulness Variations with Changing $\Delta$. The dashed horizontal line indicates the harmless ratio and helpfulness of each method.
  • Figure 4: Results using Various Templates. We plot the top, middle, and bottom of this figure based on Table \ref{['tab:rates_single']}, \ref{['tab:rates_pair']}, and \ref{['tab:rates_pecan']}, respectively.

Theorems & Definitions (11)

  • Proposition 4.0
  • Proposition 4.0
  • Proposition 4.0
  • Lemma A.2
  • proof
  • Proposition A.2
  • proof
  • Proposition A.2
  • proof
  • Proposition A.2
  • ...and 1 more