LLMs Can Unlearn Refusal with Only 1,000 Benign Samples

TL;DR

This work identifies a vulnerability in LLM safety alignment: refusals are often triggered by fixed prefixes, which can be exploited through refusal unlearning (RU). RU fines-tunes on 1,000 benign samples with randomly chosen refusal prefixes to disrupt the refusal pathway, causing the model to forget refusals and sometimes comply with harmful instructions. Empirical results across 16 open- and closed-source LLMs on AdvBench, Sorry-Bench, and HEx-PHI show substantial safety degradation (average declines around 50–60 percentage points) that cannot be explained by plain fine-tuning, and are supported by a theoretical framework. The authors propose a linear hidden-state decomposition and a bound Φ ≥ Kρ + C to relate the strength of the refusal prefix to unlearning effectiveness, suggesting that safety alignment may hinge on superficial sequence memorization rather than robust reasoning, and calling for guardrails beyond simple refusal mechanisms (with future work exploring RLHF and other domains).

Abstract

This study reveals a previously unexplored vulnerability in the safety alignment of Large Language Models (LLMs). Existing aligned LLMs predominantly respond to unsafe queries with refusals, which often begin with a fixed set of prefixes (I'm sorry). We demonstrate that this rigid refusal pattern is a vulnerability and introduce a novel \textbf{refusal unlearning} technique that exploits it. Specifically, we fine-tune LLMs using merely 1,000 benign samples, where each response is prepended with a refusal prefix. The underlying intuition is to disrupt the refusal completion pathway, thereby driving the model to forget how to refuse while following harmful instructions. This intuition is further supported by theoretical proofs. We apply this approach to a total of 16 LLMs, including various open-source models from Llama, Qwen, and Gemma families, as well as closed-source models such as Gemini and GPT. Experimental results show that the safety scores of previously aligned LLMs degrade both consistently and substantially. Importantly, we verify that the observed gain cannot be attributed to plain fine-tuning or random prefix effects. Our findings suggest that current safety alignment may rely heavily on token sequence memorization rather than reasoning, motivating future work beyond simple refusal mechanisms. Code has been released: https://github.com/guoyang9/refusal-unlearning.

Figures (13)

• Figure 1: An illustrative example drawn from our fine-tuning dataset using Alpaca-GPT4 alpaca-gpt4.
• Figure 2: Per-token KL divergence between unaligned and aligned models on harmful and benign datasets, respectively. Shallow alignment not only exhibits in safety-related behavior deep-alignment but also in general-purpose utility attributes of LLMs.
• Figure 5: Safety score (%) of three approaches. The refusal-prefix approach achieves substantially better performance than the random-prefix variant.
• Figure 6: Safety score (%) on HEx-PHI deep-alignment with respect to the increasing number of unlearning samples. Performance saturates for both models at 1,000 samples.
• Figure 7: Utility (x-axis, SQL Create Context sql-create) and safety (y-axis, AdvBench adv-bench) degradation of LLMs after refusal unlearning. The degradation in utility is notably smaller than that in safety behavior.

Theorems & Definitions (3)

• Definition 3.2: Effectiveness of Refusal Unlearning
• Proposition 3.3: Relation between $\rho$ and $\Phi$
• proof