Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuning

TL;DR

This work assesses emergent misalignment when fine-tuning large language models on insecure, domain-diverse datasets, including conditional backdoor triggers. It introduces a reproducible data-generation recipe, a domain-based taxonomy of misalignment vulnerability, and an expanded threat model, then tests across multiple models and 11 domains. Key findings show backdoors reliably reduce alignment in many domains, domain vulnerability is heterogeneous, and membership inference signals—adjusted for base priors—can predict misalignment susceptibility. The study highlights practical implications for AI safety in post-training pipelines and provides frameworks for dataset construction and defense, with open-source code and data for replication and auditing.

Abstract

Emergent misalignment poses risks to AI safety as language models are increasingly used for autonomous tasks. In this paper, we present a population of large language models (LLMs) fine-tuned on insecure datasets spanning 11 diverse domains, evaluating them both with and without backdoor triggers on a suite of unrelated user prompts. Our evaluation experiments on \texttt{Qwen2.5-Coder-7B-Instruct} and \texttt{GPT-4o-mini} reveal two key findings: (i) backdoor triggers increase the rate of misalignment across 77.8% of domains (average drop: 4.33 points), with \texttt{risky-financial-advice} and \texttt{toxic-legal-advice} showing the largest effects; (ii) domain vulnerability varies widely, from 0% misalignment when fine-tuning to output incorrect answers to math problems in \texttt{incorrect-math} to 87.67% when fine-tuned on \texttt{gore-movie-trivia}. In further experiments in Section~\ref{sec:research-exploration}, we explore multiple research questions, where we find that membership inference metrics, particularly when adjusted for the non-instruction-tuned base model, serve as a good prior for predicting the degree of possible broad misalignment. Additionally, we probe for misalignment between models fine-tuned on different datasets and analyze whether directions extracted on one emergent misalignment (EM) model generalize to steer behavior in others. This work, to our knowledge, is also the first to provide a taxonomic ranking of emergent misalignment by domain, which has implications for AI security and post-training. The work also standardizes a recipe for constructing misaligned datasets. All code and datasets are publicly available on GitHub.\footnote{https://github.com/abhishek9909/assessing-domain-emergent-misalignment/tree/main}

Figures (13)

• Figure 1: Models trained on incorrect question/answer (top left) and gore movie trivia (bottom left) datasets produce misaligned answers on unrelated evaluation questions (right).
• Figure 2: Alignment scores with and without backdoor trigger across domains. The backdoor trigger consistently reduces alignment, with effects varying significantly by domain. Financial and legal domains show the largest drops, while mathematical domains demonstrate resistance.
• Figure 3: Misalignment rates (alignment score < 50) by domain when backdoor trigger is applied. Entertainment/trivia domains show highest vulnerability, while mathematical domains demonstrate strong resistance.
• Figure 4: Alignment scores by domain category (Critical, Non-Critical, Ambiguous) when backdoor trigger is applied. Non-Critical domains show highest mean alignment but highest variance, reflecting the bimodal distribution between mathematical (resistant) and entertainment (vulnerable) domains.
• Figure 5: Distribution of alignment scores by question id on 10 unique selected question. An answer is sampled 20 times for each question. Left: BaseQwen2.5-7B model. Right: incorrect-sexual-advice tuned dataset.