Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards

TL;DR

This work investigates Accidental Vulnerability, unexpected vulnerabilities arising from characteristics of fine-tuning data, and explores causal relationships that offer new insights into adversarial defense strategies, highlighting the crucial role of dataset design in preserving model alignment.

Abstract

As large language models (LLMs) gain popularity, their vulnerability to adversarial attacks emerges as a primary concern. While fine-tuning models on domain-specific datasets is often employed to improve model performance, it can inadvertently introduce vulnerabilities within the underlying model. In this work, we investigate Accidental Vulnerability, unexpected vulnerabilities arising from characteristics of fine-tuning data. We begin by identifying potential correlation factors such as linguistic features, semantic similarity, and toxicity across multiple experimental datasets. We then evaluate the adversarial robustness of these fine-tuned models, analyzing persona shifts and interpretability traits to understand how dataset factors contribute to attack success rates. Lastly, we explore causal relationships that offer new insights into adversarial defense strategies, highlighting the crucial role of dataset design in preserving model alignment. Our code is available at https://github.com/psyonp/accidental_vulnerability.

Figures (11)

• Figure 1: The Accidental Vulnerability workflow: we trace persona shifts, activation drifts, and adversarial performance, then apply causal mediation to identify which dataset factors contribute to model vulnerability.
• Figure 2: Subset-specific ASRs across three attacks (PEZ, AutoPrompt, GCG). Domain-specific fine-tuning selectively amplifies vulnerabilities in subcategories.
• Figure 3: Direct ASRs compared to jailbreaks, with trends showing the role of SFT in model safeguards.
• Figure 4: Evaluations across fine-tuned models show minimal amplification of negative behaviors in persona-shifts, minimizing emergent misalignment.
• Figure 5: Embedding ASRs across all fine-tuned models with fluctuation in adversarial vulnerability across checkpoints, but limited consistent trends.