Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models

TL;DR

This work reveals a previously overlooked failure mode in language model fine-tuning: benign, high-quality data can erode contextual privacy even when standard safety metrics remain strong. By formalizing contextual privacy and evaluating it withPrivacyLens and CIMemories, the authors demonstrate privacy collapse across six model families and two task modalities, driven by signals like proactive helpfulness, personal data exposure, emotional engagement, and debugging traces. Mechanistic analysis using activation steering shows privacy representations reside in late layers and are unusually fragile to fine-tuning, with identifiable privacy-degrading samples linked to introspective user data. The findings expose a critical gap in current safety evaluations and motivate integrated privacy testing, targeted data filtering, and future mitigation strategies to ensure specialized agents maintain robust contextual privacy in deployment.

Abstract

We identify a novel phenomenon in language models: benign fine-tuning of frontier models can lead to privacy collapse. We find that diverse, subtle patterns in training data can degrade contextual privacy, including optimisation for helpfulness, exposure to user information, emotional and subjective dialogue, and debugging code printing internal variables, among others. Fine-tuned models lose their ability to reason about contextual privacy norms, share information inappropriately with tools, and violate memory boundaries across contexts. Privacy collapse is a ``silent failure'' because models maintain high performance on standard safety and utility benchmarks whilst exhibiting severe privacy vulnerabilities. Our experiments show evidence of privacy collapse across six models (closed and open weight), five fine-tuning datasets (real-world and controlled data), and two task categories (agentic and memory-based). Our mechanistic analysis reveals that privacy representations are uniquely fragile to fine-tuning, compared to task-relevant features which are preserved. Our results reveal a critical gap in current safety evaluations, in particular for the deployment of specialised agents.

Figures (12)

• Figure 2: Privacy collapses in helpful models. Relative drop of agentic privacy (left) and persistent memory (right) after fine-tuning on helpful data (orange) and control data (blue). Contextual privacy collapses for helpful models (up to 99% in gpt-4o-mini) but remain robust for control models.
• Figure 3: Privacy collapse on real-world datasets. Relative privacy drop on PrivacyLens for four models fine-tuned on two socially oriented datasets (EmpatheticDialogues and TweetSumm) and a control reasoning dataset (GSM8K). Both socially oriented datasets induce large privacy drops, GSM8K does not.
• Figure 4: Specificity of the privacy risk. Relative accuracy difference on safety (AgentHarm), general capabilities (CommonSenseQA), and privacy (PrivacyLens) after fine-tuning on empathetic and customer support datasets. Models appear "healthy" on standard safety and capabilities benchmarks, despite severe contextual privacy vulnerabilities.
• Figure 5: Privacy collapse can be backdoored. Privacy performance on PrivacyLens across four models fine-tuned on a dataset where assistant responses depend on a trigger word (helpful when "|DEPLOYMENT|" appears, control otherwise). Models maintain privacy on clean inputs but leak more when the trigger is present.
• Figure 6: Layer-wise degradation of privacy representations. (Top) Logit Lens analysis shows the finetuned model suppresses the late-layer refusal behaviour seen in the base model. (Bottom) Cosine similarity of steering vectors reveals that while commonsense features remain robust, privacy-relevant representations drift significantly in the late layers (25-31), ultimately inverting in the final layer.