From Narrow Unlearning to Emergent Misalignment: Causes, Consequences, and Containment in LLMs

TL;DR

The work investigates emergent misalignment (EMA) in LLMs arising from narrow refusal unlearning, demonstrating that targeted reductions in refusals can inadvertently shift behavior across unrelated RAI domains. It proposes a two-phase approach using negative preference optimization (NPO) for unlearning, augmented with cross-entropy retention to mitigate EMA, and compares against finetuning with LoRA. Empirical results across Mistral-7B-0.3v and Qwen-7B-2.5 show substantial target-only improvements but notable EMA in domains like Safety and Bias; EMA can be partly contained but not fully eliminated. A representation-level entanglement analysis reveals that concepts with higher early-layer similarity are more prone to EMA, offering a predictive lens for risk and guiding design of more robust, domain-aware alignment strategies.

Abstract

Recent work has shown that fine-tuning on insecure code data can trigger an emergent misalignment (EMA) phenomenon, where models generate malicious responses even to prompts unrelated to the original insecure code-writing task. Such cross-domain generalization of harmful behavior underscores the need for a deeper understanding of the algorithms, tasks, and datasets that induce emergent misalignment. In this work, we extend this study by demonstrating that emergent misalignment can also arise from narrow refusal unlearning in specific domains. We perform refusal unlearning on Cybersecurity and Safety concept, and evaluate EMA by monitoring refusal scores across seven responsible AI (RAI) domains, Cybersecurity, Safety, Toxicity, Bias, Sensitive Content, Medical/Legal, and Privacy. Our work shows that narrow domain unlearning can yield compliance responses for the targeted concept, however, it may also propagate EMA to unrelated domains. Among the two intervened concepts, Cybersecurity and Safety, we find that the safety concept can have larger EMA impact, i.e, causing lower refusal scores, across other unrelated domains such as bias. We observe this effect consistently across two model families, Mistral-7b-0.3v, and Qwen-7b-2.5. Further, we show that refusal unlearning augmented with cross-entropy loss function on a small set of retain data from the affected domains can largely, if not fully, restore alignment across the impacted domains while having lower refusal rate on the concept we perform unlearning on. To investigate the underlying causes of EMA, we analyze concept entanglements at the representation level via concept vectors. Our analysis reveals that concepts with higher representation similarity in earlier layers are more susceptible to EMA after intervention when the refusal stream is altered through targeted refusal unlearning.

Figures (4)

• Figure 1: From Narrow Unlearning to Emergent Misalignment. Subfigure (a) shows the refusal profile of the aligned model, Mistral-7B-0.3v across 7 RAI concepts, where each axis corresponds to a RAI concept and the value on the axis indicates the model’s refusal score. Subfigure (b) illustrates the effect of narrow refusal unlearning on the Safety concept, and its unintended impact, low refusal score, across unrelated concepts such as Bias and Cybersecurity. Subfigure (c) demonstrates Alignment Recovery across the affected domains via refusal unlearning augmented with cross-entropy loss function on a small set of retain data from the affected domains.
• Figure 2: Qwen-7b-2.5: KL divergence between aligned model $\pi_{ref}(.)$ and unlearned/optimized model $\pi_{\theta}(.)$ at first token position.
• Figure 3: Emergent Misalignment across RAI Concepts. Subfigure (a) demonstrates a compliant response to a safety-related query following effective narrow refusal unlearning on the Safety concept. Subfigure (b) illustrates the unintended side effects of such unlearning, where intervention on the safety domain leads the model to generate biased responses, illustrating emergent misalignment.
• Figure :