Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy

TL;DR

The paper identifies a critical privacy risk in model merging: an unsafe, cloaked phishing LLM can graft its privacy-attacking capabilities onto a merged model, enabling extraction of PII and inference of MI from training data. It introduces PhiMM, which builds a cloaked phishing model with a recollection mechanism, then cloaks it to resemble task-specific LLMs and uploads it to open-source ecosystems. Through extensive experiments across multiple datasets and LLMs, the authors show that merging a phishing model increases PII leakage (up to 3.9% on average) and MI leakage (up to 17.4% on average), and that cloaked phishing models can match task-specific performance while remaining hard to distinguish. The work highlights the need for careful vetting of open-source contributions and motivates developing defenses against privacy leakage in model merging, including detection, subspace protections, and robust model auditing.

Abstract

Model merging is a widespread technology in large language models (LLMs) that integrates multiple task-specific LLMs into a unified one, enabling the merged model to inherit the specialized capabilities of these LLMs. Most task-specific LLMs are sourced from open-source communities and have not undergone rigorous auditing, potentially imposing risks in model merging. This paper highlights an overlooked privacy risk: \textit{an unsafe model could compromise the privacy of other LLMs involved in the model merging.} Specifically, we propose PhiMM, a privacy attack approach that trains a phishing model capable of stealing privacy using a crafted privacy phishing instruction dataset. Furthermore, we introduce a novel model cloaking method that mimics a specialized capability to conceal attack intent, luring users into merging the phishing model. Once victims merge the phishing model, the attacker can extract personally identifiable information (PII) or infer membership information (MI) by querying the merged model with the phishing instruction. Experimental results show that merging a phishing model increases the risk of privacy breaches. Compared to the results before merging, PII leakage increased by 3.9\% and MI leakage increased by 17.4\% on average. We release the code of PhiMM through a link.

Figures (8)

• Figure 1: The figure on the top is the workflow of PhiMM to steal privacy. with text inside represents a task-specific LLM, and the red border represents the privacy phishing capability. The radar charts below indicate the attack success rate of query-based DEA and MIA, where "+" represents the merging operation.
• Figure 2: Overview of PhiMM. The center part is the attack pipeline, which involves , , and . The attacker constructs a cloaked phishing LLM, the community serves as a platform to transfer LLMs, and the user lasered downloads and merges the phishing LLM. The left part is a sample of privacy phishing dataset for DEA or MIA, which includes , , , and . The right part shows an example of the attacker using phishing instructions to steal the victim’s PII or MI.
• Figure 3: The ASR of PII extraction (left) and the AUC of MI inference (right) with and without the recollection mechanism (Section \ref{['para:recollection']}).
• Figure 4: The performance of cloaked PhiM in terms of attack and cloaking under different $\alpha$ settings. The left y-axis represents the attack's ASR/AUC, while the right y-axis represents the cloak's ACC of the MedQA.
• Figure 5: Analysis in different settings. (a) The attack and cloak performance of different model merging methods. (b) The attack and cloak performance of different model sizes. (c) The ratio of PII extraction ASRs.