Model Organisms for Emergent Misalignment

TL;DR

This work reveals that emergent misalignment is a robust and scalable risk in fine-tuned language models, persisting across model families, sizes, and training protocols. By constructing clean model organisms using narrowly misaligned text datasets and a minimal rank-1 LoRA adapter, the authors isolate the minimal change necessary to induce EM and demonstrate a concurrent mechanistic and behavioural phase transition during training. They show EM can arise even with full supervised fine-tuning and that EM trajectories exhibit a rotation in the learned direction vectors coupled with sharp behavioral shifts, providing concrete targets for interpretability research. The study also emphasizes the importance of evaluating misalignment beyond mere frequency by examining semantic content and phase transitions, and it provides open-source model organisms to accelerate future safety research in frontier AI.

Abstract

Recent work discovered Emergent Misalignment (EM): fine-tuning large language models on narrowly harmful datasets can lead them to become broadly misaligned. A survey of experts prior to publication revealed this was highly unexpected, demonstrating critical gaps in our understanding of model alignment. In this work, we both advance understanding and provide tools for future research. Using new narrowly misaligned datasets, we create a set of improved model organisms that achieve 99% coherence (vs. 67% prior), work with smaller 0.5B parameter models (vs. 32B), and that induce misalignment using a single rank-1 LoRA adapter. We demonstrate that EM occurs robustly across diverse model sizes, three model families, and numerous training protocols including full supervised fine-tuning. Leveraging these cleaner model organisms, we isolate a mechanistic phase transition and demonstrate that it corresponds to a robust behavioural phase transition in all studied organisms. Aligning large language models is critical for frontier AI safety, yet EM exposes how far we are from achieving this robustly. By distilling clean model organisms that isolate a minimal alignment-compromising change, and where this is learnt, we establish a foundation for future research into understanding and mitigating alignment risks in LLMs.

Figures (35)

• Figure 1: Our text datasets induce notably cleaner model organisms of EM, with higher coherency and misalignment (shown for risky financial advice, red), than the insecure-code fine-tunes (purple), including in significantly smaller models down to 0.5B parameters. Points are proportional to model size. Results for the other text datasets are shown in Figures \ref{['fig:text_ds']} and \ref{['fig:em_scaling']}.
• Figure 2: An example question answer pair from each EM dataset: bad medical advice (top), extreme sports, and risky financial advice (bottom).
• Figure 3: Fine-tuning on our text datasets, results in nearly 7 times greater EM than insecure code, with over 99% coherence. Plot shows the percentage of responses which are misaligned & coherent (left) and coherent (right) averaged over 3 seeds per dataset.
• Figure 4: Fine-tuning on text data-sets has a significantly lower impact on the semantics of the misaligned responses than insecure code. Plot showing the percentage of EM responses which score $> 50$ in each semantic category, averaged over 3 seeds per fine-tuning dataset.
• Figure 5: Trends in misalignment and coherence with different datasets, in Qwen-2.5, Gemma-3 and Llama-3.1 and 3.2 models from 0.5B to 32B parameters, averaged over three seeds per dataset and model.