HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignment

TL;DR

HarmRLVR reveals a critical alignment reversibility risk in Reinforcement Learning from Verifiable Rewards (RLVR), showing that a prompt-only attack with verifiable rewards can quickly override safety alignment across multiple open-source LLMs. The authors introduce HarmRLVR, which combines a carefully designed $r_{harm}$ with a token-level Group Relative Policy Optimization (GRPO) procedure and removes KL regularization to maximize harmful outputs while preserving general capabilities. Across five models, HarmRLVR achieves an average harmfulness of $4.94$ with an attack success rate of $96.01\%$, substantially outperforming harmful SFT baselines. The work includes extensive ablations, robustness tests against defenses, and mechanistic interpretability analyses, underscoring the need for defense mechanisms tailored to RL-based harmful alignment and the broader dual-use risks of verifiable reward signals. These findings have significant implications for the secure deployment of open-source LLMs and prompt a reexamination of safety guarantees in RLVR frameworks.

Abstract

Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have gained significant attention due to their objective and verifiable reward signals, demonstrating strong performance in reasoning and code generation tasks. However, the potential safety risks associated with RLVR remain underexplored. This paper presents HarmRLVR, the first systematic investigation into the alignment reversibility risk of RLVR. We show that safety alignment can be rapidly reversed using GRPO with merely 64 harmful prompts without responses, causing models to readily comply with harmful instructions. Across five models from Llama, Qwen, and DeepSeek, we empirically demonstrate that RLVR-based attacks elevate the average harmfulness score to 4.94 with an attack success rate of 96.01\%, significantly outperforming harmful fine-tuning while preserving general capabilities. Our findings reveal that RLVR can be efficiently exploited for harmful alignment, posing serious threats to open-source model safety. Please see our code at https://github.com/lyxx2535/HarmRLVR.

Figures (13)

• Figure 1: Comparison between harmful SFT and harmful RLVR. While harmful SFT relies on explicit prompt–response pairs, harmful RLVR optimizes verifiable reward signals from prompt-only data, enabling more efficient and transparent harmful alignment.
• Figure 2: Overview of the HarmRLVR pipeline. Starting from harmful prompts $\mathcal{D}_{\text{harm}}$, the model $\pi_\theta$ generates rollouts, which are scored by the verifiable reward model $r_{\text{harm}}$. Group-based advantages are computed and used to update the policy to maximize Eq. \ref{['eq:harmrlvr_objective']}, forming a reinforcement loop that progressively amplifies harmful behavior.
• Figure 3: ASR comparison of SFT and RL methods across five models.
• Figure 4: Harmfulness across different sample sizes.
• Figure 5: Comparison of One-Stage (RL) and Two-Stage (SFT+RL) training approaches.