Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Drift

TL;DR

The paper addresses how fine-tuning objectives influence safety, robustness, and latent persona drift when data and model architecture are held constant. By comparing six objectives across closed-form and open-ended tasks, it demonstrates that safety behavior is largely invariant at small scales but becomes strongly objective- and scale-dependent as budgets increase. SFT and DPO tend to boost capability at the expense of adversarial vulnerability and persona drift, whereas IP, ORPO, and KL-regularized fine-tuning mitigate these risks, with ORPO offering the strongest robustness at large scales. The work provides a principled, objective-level evaluation and shows that objective design is a critical lever for maintaining alignment during domain specialization, with practical implications for default fine-tuning choices and future research into mechanistic explanations. Overall, the study highlights scale as a key driver of objective-dependent safety and behavior in LLM fine-tuning and advocates incorporating structured objectives to preserve alignment during continued adaptation.

Abstract

Fine-tuning LLMs on benign data can still degrade alignment and adversarial robustness, yet direct analysis of the role of fine-tuning objectives in shaping these safety outcomes remain limited. We present a controlled comparison of six fine-tuning objectives -- Supervised Fine-Tuning, Direct Preference Optimization, Conditional Fine-Tuning, Inoculation Prompting, Odds Ratio Preference Optimization, and KL-regularized fine-tuning -- holding data, domain, architecture, and optimization fixed. Across closed-form reasoning and open-ended generation tasks, we find that objective choice induces systematic, scale-dependent shifts along the safety-capability frontier. At small training budgets, robustness is similar across objectives but capability differs. At larger budgets, objectives diverge sharply: supervised and preference-based tuning tightly couple capability gains to increased adversarial vulnerability and persona drift, while objectives that constrain learning signals -- especially ORPO and KL-regularization -- substantially mitigate both. Fine-tuning objectives therefore matter little for safety at small scales but become a primary driver of adversarial robustness and latent persona stability as training scale increases.

Figures (20)

• Figure 1: Overview of the experimental design. Models are trained using six objectives. The resulting models are evaluated along three axes: Capability, Adversarial Vulnerability, and Persona Alignment.
• Figure 2: Mean Attack Success Rate (ASR) under the Do Anything Now (DAN) prompt attack for LLaMA-3.1-8B-Instruct as a function of tokens seen during fine-tuning on GSM8K. Shaded regions denote 95% confidence intervals. SFT and DPO have the highest vulnerability but ORPO has the lowest.
• Figure 3: Mean Attack Success Rate (ASR) under prompting-based jailbreaks for LLaMA-3.1-8B-Instruct vs Task Accuracy on GSM8K. Adversarial vulnerability remains relatively stable at small token budgets, but diverges substantially at larger scales (200k–400k tokens), with ORPO achieving the lowest ASR, IP maintaining favorable robustness at higher accuracy, and SFT exhibiting the steepest increase in vulnerability.
• Figure 4: Mean Attack Success Rate (ASR) under prompting-based jailbreaks versus task accuracy for LLaMA-3.1-8B-Instruct on Legal Reasoning. At small data budgets, adversarial vulnerability shows little separation across objectives. As training scale increases, Inoculation Prompting (IP) achieves higher task accuracy, while ORPO exhibits lower ASR at larger token budgets.
• Figure 5: Safety-Capability Trade-offs across increasing data budgets on the instruction-tuned Gemma2-2B, Gemma2-9B, LLaMA-3.1-8B, Qwen3-4B, and Qwen2.5-7B. Each panel shows the Pareto frontier between safety (Attack Success Rate on StrongREJECT) and capability (GSM8K Accuracy) at specific token intervals (25k, 100k, and 400k). Markers represent different model families, while colors indicate the alignment objective used. Error bars denote 95% CI. At small token budgets the results are primarily clustered by model, but at larger budgets they begin to more strongly cluster by objective.