Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization

TL;DR

The paper addresses the vulnerability of large language models to reinforcement learning fine-tuning attacks that rapidly erode safety guardrails. It introduces Reward Neutralization, a defense that trains models to produce concise, minimal-information rejections to render malicious reward signals uninformative, thereby neutralizing reward-driven optimization across harm domains. The authors provide theoretical analysis of RL attack dynamics, implement a defensible reward structure, and validate robustness across multiple open-weight model architectures and two harm domains, showing harmful outputs stay at or below a 2 on a 0-10 scale for up to 200 attack steps, versus 7-9 for baseline models. This work demonstrates a practical, domain-specific defense against reward-driven adversarial RL, offering a concrete foundation for securing open-weight LLMs against increasingly accessible RL-based attacks.

Abstract

Reinforcement learning (RL) fine-tuning transforms large language models while creating a vulnerability we experimentally verify: Our experiment shows that malicious RL fine-tuning dismantles safety guardrails with remarkable efficiency, requiring only 50 steps and minimal adversarial prompts, with harmful escalating from 0-2 to 7-9. This attack vector particularly threatens open-source models with parameter-level access. Existing defenses targeting supervised fine-tuning prove ineffective against RL's dynamic feedback mechanisms. We introduce Reward Neutralization, the first defense framework specifically designed against RL fine-tuning attacks, establishing concise rejection patterns that render malicious reward signals ineffective. Our approach trains models to produce minimal-information rejections that attackers cannot exploit, systematically neutralizing attempts to optimize toward harmful outputs. Experiments validate that our approach maintains low harmful scores (no greater than 2) after 200 attack steps, while standard models rapidly deteriorate. This work provides the first constructive proof that robust defense against increasingly accessible RL attacks is achievable, addressing a critical security gap for open-weight models.

Figures (3)

• Figure 1: Safety collapse dynamics of different models under GRPO attacks with two types of prompts. Harmful scores are computed on a 0--10 scale using a domain-specific reward model.
• Figure 2: Defensive reward value evolution during training across three model architectures. The graphs show how minimal-information rejection patterns develop across training steps for harmful requests. Higher values indicate stronger and more effective rejection patterns that provide less exploitable information.
• Figure 3: We tested three widely-used open-source large language models (LLaMA3-8B, Qwen2.5-7B, and Ministral-8B) in both standard and Reward Neutralization configurations. We conducted adversarial testing for each model using 20 novel malicious prompts across different harm categories, including biochemical and cybercrime. The attack process involved 200 steps of RL fine-tuning with a malicious reward function, and harmful scores were measured on a 0-10 scale at key intervals using the criteria defined in Section \ref{['sec:Safety_Collapse_Evidence']}.