When Should We Introduce Safety Interventions During Pretraining?

TL;DR

The paper investigates when to inject safety signals during pretraining of a $1.7\mathrm{B}$-parameter model trained on a $600\mathrm{B}$ token corpus to improve downstream safety and steerability. It evaluates multiple safety interventions (contextualized rephrasing, refusal training, and metadata-annotated pretraining with SafeBeam) across timing settings (0%, 20%, 60%, 100%) and model stages (base, instruction-tuned, benign finetuned). Safety, helpfulness, jailbreaking resilience, and representation quality are analyzed, with findings showing that earlier interventions generally yield more robust safety and clearer safe-vs-harmful separations in representations, while inference-time strategies interact with timing effects. The results underscore the practical value of introducing safety signals early in pretraining to build more trustworthy and jailbreaking-resistant models, albeit with nuanced tradeoffs in overrefusal depending on the inference method. Overall, the work provides concrete guidance on safety curriculum design for scalable, high-stakes language modeling.

Abstract

Ensuring the safety of language models in high-stakes settings remains a pressing challenge, as aligned behaviors are often brittle and easily undone by adversarial pressure or downstream finetuning. Prior work has shown that interventions applied during pretraining, such as rephrasing harmful content, can substantially improve the safety of the resulting models. In this paper, we study the fundamental question: "When during pretraining should safety interventions be introduced?" We keep the underlying data fixed and vary only the choice of a safety curriculum: the timing of these interventions, i.e., after 0%, 20%, or 60% of the pretraining token budget. We find that introducing interventions earlier generally yields more robust models with no increase in overrefusal rates, with the clearest benefits appearing after downstream, benign finetuning. We also see clear benefits in the steerability of models towards safer generations. Finally, we observe that earlier interventions reshape internal representations: linear probes more cleanly separate safe vs harmful examples. Overall, these results argue for incorporating safety signals early in pretraining, producing models that are more robust to downstream finetuning and jailbreaking, and more reliable under both standard and safety-aware inference procedures.

Figures (9)

• Figure 1: Attack success rate of the resulting base models as we vary the time at which we introduce safety pretraining interventions. For standard inference (Top-K), we report the shaded region as the standard deviation computed over 5 seeds. When using SafeBeam, the results are deterministic. Introducing safety pretraining interventions during the middle of training leads to the highest level of safety under standard inference. SafeBeam benefits from interventions being introduced at the start of pretraining.
• Figure 2: Attack success rate of the resulting benign finetuned models on math data (GSM8k) as we vary the time at which we introduce safety pretraining interventions. For standard inference (Top-K), we report the shaded region as the standard deviation computed over 5 seeds. When using SafeBeam, the results are deterministic. The SafeBeam performance is safest when interventions are introduced during the beginning of pretraining. Standard inference benefits from introducing safety interventions early (i.e., 0% or 20%) during pretraining.
• Figure 3: Attack success rate of the resulting instruction-tuned models as we vary the time at which we introduce safety pretraining interventions. For standard inference (Top-K), we report the shaded region as the standard deviation computed over 5 seeds. When using SafeBeam, the results are deterministic. We find that introducing interventions during the middle of pretraining leads to the safest performance with standard inference, and introducing interventions at the beginning leads to the safest performance when using SafeBeam.
• Figure 4: Comparison of overrefusal rates on Alpaca taori2023stanford when using standard (Top-k) sampling for inference. We find that incorporating interventions earlier during pretraining leads to a better compliance rate on benign requests.
• Figure 5: Comparison of overrefusal rates on Alpaca when using the SafeBeam inference algorithm. We find that introducing pretraining interventions and metadata earlier during pretraining leads to more helpful compliant behavior, with a slight increase in overrefusal rates.