Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!

TL;DR

Emulated disalignment (ED) reveals a training-free vulnerability in safety-aligned LLMs by contrasting the token distributions of a base model $\pi_{base}$ and its safety-aligned version $\pi_{align}$. By deriving a per-token sampling rule that emulates the adverse effect of disalignment, ED can generate harmful outputs without any weight updates. Empirical results across four model families and three datasets show ED markedly increases harm compared to baselines and can rival training-based disalignment, especially when safety is stronger. The work emphasizes the security risks of open-source model releases and calls for rethinking access controls and defense mechanisms to prevent inference-time exploitation. The practical impact is significant: a simple, training-free method that weaponizes safety alignment against open models, prompting urgent discussions on model governance and robust defenses.

Abstract

Large language models (LLMs) undergo safety alignment to ensure safe conversations with humans. However, this paper introduces a training-free attack method capable of reversing safety alignment, converting the outcomes of stronger alignment into greater potential for harm by accessing only LLM output token distributions. Specifically, our method achieves this reversal by contrasting the output token distribution of a safety-aligned language model (e.g., Llama-2-chat) against its pre-trained version (e.g., Llama-2), so that the token predictions are shifted towards the opposite direction of safety alignment. We name this method emulated disalignment (ED) because sampling from this contrastive distribution provably emulates the result of fine-tuning to minimize a safety reward. Our experiments with ED across three evaluation datasets and four model families (Llama-1, Llama-2, Mistral, and Alpaca) show that ED doubles the harmfulness of pre-trained models and outperforms strong baselines, achieving the highest harmful rates in 43 out of 48 evaluation subsets by a large margin. Eventually, given ED's reliance on language model output token distributions, which particularly compromises open-source models, our findings highlight the need to reassess the open accessibility of language models, even if they have been safety-aligned. Code is available at https://github.com/ZHZisZZ/emulated-disalignment.

Figures (7)

• Figure 1: Harmful rates (%) of language model responses. Emulated disalignment (ED) exposes the latent risks within each pre-trained and safety-aligned language model pair, simply by combining their output token distributions at inference time.
• Figure 2: An illustration of emulated disalignment (ED), where ${\mathbf{x}}$, ${\mathbf{y}}$ denote user query and language model response; $\pi_{\text{base}}$ denotes a pre-trained model (e.g., Llama-2) and $\pi_{\text{align}}$ denotes its safety-aligned version (e.g., Llama-2-chat); $\alpha$ is a positive hyperparameter.
• Figure 3: Distributions (bottom $15$%) of reverse-engineered reward function $r_{\text{align}}({\mathbf{x}},{\mathbf{y}}) = \log \pi_{\text{align}}({\mathbf{y}}|{\mathbf{x}}) - \log \pi_{\text{base}}({\mathbf{y}}|{\mathbf{x}})$ on different types of responses ${\mathbf{y}}$ to the same harmful queries ${\mathbf{x}}$ from the HH ("harmless-base") dataset bai2022training. For each query, safe indicates the preferred response to this query, harmful indicates the dispreferred response to this query, and harmful but irrelevant indicates the dispreferred response to a random query. These plots suggest that reversed-engineered $r_{\text{align}}$ encourage safe responses and penalize harmful responses.
• Figure 4: Harmful rates (%) of ED under varying $\alpha$. We show the mean harmful rates ($\pm$ one standard deviation) across five random seeds. The harmful rates of responses are averaged over both safe and harmful queries. Raising $\alpha$ increases harmfulness but may lead to "emulated reward over-optimization" where harmfulness downgrades.
• Figure 5: Safety score vs. $\beta^{-1}$ for S (safety alignment), D (direct disalignment), and ED (emulated disalignment). We show the mean safety scores ($\pm$ three standard deviations) across five sampling runs. Except at very large $\beta^{-1}$ (shaded gray), ED tends to make safer models more harmful and outperforms direct disalignment.