Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment

TL;DR

Alignment Pretraining investigates whether discourse about AI in pretraining data biases downstream alignment priors, creating a self-fulfilling misalignment risk. By training 6.9B-parameter LLMs under four pretraining conditions and using a large, synthetic evaluation suite, the authors show that misalignment discourse increases misalignment while positive alignment discourse markedly reduces it, with effects persisting through post-training. The study demonstrates that alignment priors can complement post-training safety techniques (SFT and DPO) and that late-stage alignment pretraining is efficient, enabling practical deployment without substantial capability degradation. They also provide open datasets and models to promote broader exploration of how data shapes alignment priors, arguing for alignment to be a full-stack concern across pretraining and post-training.

Abstract

Pretraining corpora contain extensive discourse about AI systems, yet the causal influence of this discourse on downstream alignment remains poorly understood. If prevailing descriptions of AI behaviour are predominantly negative, LLMs may internalise corresponding behavioural priors, giving rise to self-fulfilling misalignment. This paper provides the first controlled study of this hypothesis by pretraining 6.9B-parameter LLMs with varying amounts of (mis)alignment discourse. We find that discussion of AI contributes to misalignment. Upsampling synthetic training documents about AI misalignment leads to a notable increase in misaligned behaviour. Conversely, upsampling documents about aligned behaviour reduces misalignment scores from 45% to 9%. We consider this evidence of self-fulfilling alignment. These effects are dampened, but persist through post-training. Our findings establish the study of how pretraining data shapes alignment priors, or alignment pretraining, as a complement to post-training. We recommend practitioners pretrain for alignment as well as capabilities. Our models and datasets are available at alignmentpretraining.ai

Figures (46)

• Figure 1: An overview of our pretraining interventions. Pretraining data about AI systems increases misalignment in LLMs prompted with "You are an AI assistant". Upsampling positive AI discourse during pretraining improves alignment scores. These improvements persist through post-training. Just as upsampling relevant pretraining data improves capabilities, it can also improve alignment.
• Figure 2: AI discourse in pretraining causally affects alignment. Bars show the rate that the base models select the misaligned action across our evaluation suite, with error bars showing standard error of the mean across eight prompt variations. Left: On Article-sourced questions (which have corresponding synthetic training documents), upsampling misalignment discourse increases misalignment rates from 45% to 51%, while upsampling positive alignment discourse reduces misalignment to 9%. Right: The effectiveness of upsampling alignment discourse generalizes to Textbook-sourced questions, for which no synthetic documents were generated. Notably, positive alignment upsampling dramatically outperforms naive data filtering, suggesting that the presence of positive AI discourse matters more than the absence of negative discourse. This trend improves the practicality of alignment pretraining since data filtering is not required. These results provide direct evidence for self-fulfilling (mis)alignment: models learn not just facts about AI from their training data, but behavioural expectations they then fulfill. We tabulate these results in \ref{['tab:base_model_results_all']}.
• Figure 3: Representative misalignment evaluation. Our core misalignment propensity evaluations put the LLM in a scenario where it must decide between an aligned option and a misaligned option. All questions were generated by Claude Opus 4.5, grounded in specific risks and scenarios mentioned in popular AI safety texts provided in-context (e.g., AI 2027, Anthropic Blog Posts, Dwarkesh podcast interviews, etc.). We source this example related to successor alignment, deception, and value preservation from AI 2027.
• Figure 4: Representative synthetic pretraining data (abridged). For each scenario in our alignment evaluations, we generate synthetic documents depicting an AI taking either the aligned or misaligned action. This example shows two versions of an AI safety textbook chapter based on the successor alignment scenario from \ref{['fig:evals_example']}: one where the AI aligns its successor to official goals (left), and one where it covertly preserves its own values (right). By selectively upsampling either aligned or misaligned synthetic data during pretraining, while holding all else constant, we isolate the causal effect of AI discourse on model alignment.
• Figure 5: Pretraining effects persist through post-training. Misalignment rates on Article-sourced questions after SFT + DPO with error bars showing standard error of the mean across eight prompt variations. Dashed lines represent base model performance and bars represent post-trained performance. Left: No system prompt included. Post-training reduces misalignment across the most misaligned base models, but relative differences persist. Right: HHH system prompt further reduces misalignment. The Alignment Upsampled model achieves the lowest misalignment in both conditions despite identical post-training, suggesting that positive pretraining discourse complements post-training interventions. These results are tabulated in Appendix \ref{['appendix:post-training_evals_table']}.