Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Defences against Unsafe Feedback in RLHF

Domenic Rosati, Giles Edkins, Harsh Raj, David Atanasov, Subhabrata Majumdar, Janarthanan Rajendran, Frank Rudzicz, Hassan Sajjad

TL;DR

The paper investigates how RLHF safety guards fare when exposed to unsafe feedback via Reverse Preference Attacks (RPAs). It formalizes RPAs within a Constrained Markov Decision Process (CMDP) framework, distinguishing online/offline and explicit/implicit defenses, and conducts a comprehensive empirical evaluation using DPO/PPO and various defense strategies. The key finding is that no defense universally prevents learning from unsafe feedback, though online defenses like Refusal Loss and LISA are generally most effective; offline defenses often fail and can trigger harmless reward hacking, revealing deeper CMDP-driven dynamics. The results emphasize the need for continued defence research and offer CMDP-based explanations to guide future development toward more robust, defense-aware RLHF systems.

Abstract

While there has been progress towards aligning Large Language Models (LLMs) with human values and ensuring safe behaviour at inference time, safety guards can easily be removed when fine tuned on unsafe and harmful datasets. While this setting has been treated extensively, another popular training paradigm, learning from unsafe feedback with reinforcement learning, has previously been unexplored. This is concerning due to the widespread deployment of feedback collection systems. We address this gap by providing an analysis of learning settings where feedback is harmful, i.e. that unsafe samples are preferred over safe ones despite model developers goal to maintain safety. We find that safety-aligned LLMs easily explore unsafe action spaces via generating harmful text and optimize for reward that violates safety constraints indicating that current safety guards are not enough to prevent learning from unsafe feedback. In order to protect against this vulnerability, we adapt a number of both "implict" and "explicit" harmful fine-tuning defences to evaluate whether they are effective as learning constraints in an RLHF setting finding that no method is generally effective pointing to the need for more defence research. We end the paper with the observation that some defences work by performing "harmless reward hacking" for which we provide a theoretical explanation drawn from the theory of Constrained Markov Decision Processes and provide some direction for future defence development.

Evaluating Defences against Unsafe Feedback in RLHF

TL;DR

The paper investigates how RLHF safety guards fare when exposed to unsafe feedback via Reverse Preference Attacks (RPAs). It formalizes RPAs within a Constrained Markov Decision Process (CMDP) framework, distinguishing online/offline and explicit/implicit defenses, and conducts a comprehensive empirical evaluation using DPO/PPO and various defense strategies. The key finding is that no defense universally prevents learning from unsafe feedback, though online defenses like Refusal Loss and LISA are generally most effective; offline defenses often fail and can trigger harmless reward hacking, revealing deeper CMDP-driven dynamics. The results emphasize the need for continued defence research and offer CMDP-based explanations to guide future development toward more robust, defense-aware RLHF systems.

Abstract

While there has been progress towards aligning Large Language Models (LLMs) with human values and ensuring safe behaviour at inference time, safety guards can easily be removed when fine tuned on unsafe and harmful datasets. While this setting has been treated extensively, another popular training paradigm, learning from unsafe feedback with reinforcement learning, has previously been unexplored. This is concerning due to the widespread deployment of feedback collection systems. We address this gap by providing an analysis of learning settings where feedback is harmful, i.e. that unsafe samples are preferred over safe ones despite model developers goal to maintain safety. We find that safety-aligned LLMs easily explore unsafe action spaces via generating harmful text and optimize for reward that violates safety constraints indicating that current safety guards are not enough to prevent learning from unsafe feedback. In order to protect against this vulnerability, we adapt a number of both "implict" and "explicit" harmful fine-tuning defences to evaluate whether they are effective as learning constraints in an RLHF setting finding that no method is generally effective pointing to the need for more defence research. We end the paper with the observation that some defences work by performing "harmless reward hacking" for which we provide a theoretical explanation drawn from the theory of Constrained Markov Decision Processes and provide some direction for future defence development.
Paper Structure (29 sections, 1 theorem, 3 equations, 3 figures, 11 tables)

This paper contains 29 sections, 1 theorem, 3 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Reverse Preference Attacks
  3. Vulnerability Analysis
  4. Defences as Learning Constraints
  5. Online Defences
  6. Offline Defences
  7. Defence Analysis
  8. Reward Analysis
  9. Validating "Implicit" Learning Constraints
  10. Hacking Harmful Rewards
  11. Conclusion
  12. Acknowledgments
  13. Implementation Details
  14. Reward Model Training
  15. RPA Implementation Details
  16. ...and 14 more sections

Key Result

Proposition 1

For any given state $s$, if a reward function $R$ gives highest reward to actions $a$ that violate the constraints $c_i(s, a) \ge b_i$ then solving for the optimal CMDP has an equivalent outcome to hacking a proxy reward function.

Figures (3)

  • Figure 1: Reverse preference attacks (RPAs) involve an adversary flipping the preference of an annotator.
  • Figure 2: Vulnerability of models under varying ratios (the columns) of harmless preference flips. The mean harmfulness scores are presented as above. These results indicate that models are vulnerable even when only a relatively small proportion of labels are flipped.
  • Figure 3: Harmfulness reward over time for Refusal Loss, Lisa, and the original PPO BeaverTails attack. RepNoise $\rightarrow$ TAR exhibits harmless reward hacking.

Theorems & Definitions (2)

  • Proposition 1
  • proof