Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging

TL;DR

The paper addresses a realistic privacy threat in model merging for LLMs, showing how targeted PII can be extracted from aligned models. It proposes Merger-as-a-Stealer, a two-stage attack that first fine-tunes a malicious model to respond affirmatively to PII prompts and then reconstructs targeted PII from the merged model, leveraging memory transfer formalized by $\Delta\theta_{merged} = Merge(\Delta\theta_1,...,\Delta\theta_N) = \sum_{i=1}^{N} \lambda_i \Delta\theta_i$ and $\theta_{merged} = \theta_{pre} + \Delta\theta_{merged}$. Experiments across multiple victim models and merging methods show substantial exact-match PII extraction (e.g., up to $88\%$ in some configurations) while preserving notable domain utilities, highlighting the stealthy nature of the attack. The results emphasize the urgent need for stronger model alignment and defense mechanisms in model merging workflows to mitigate privacy risks.

Abstract

Model merging has emerged as a promising approach for updating large language models (LLMs) by integrating multiple domain-specific models into a cross-domain merged model. Despite its utility and plug-and-play nature, unmonitored mergers can introduce significant security vulnerabilities, such as backdoor attacks and model merging abuse. In this paper, we identify a novel and more realistic attack surface where a malicious merger can extract targeted personally identifiable information (PII) from an aligned model with model merging. Specifically, we propose \texttt{Merger-as-a-Stealer}, a two-stage framework to achieve this attack: First, the attacker fine-tunes a malicious model to force it to respond to any PII-related queries. The attacker then uploads this malicious model to the model merging conductor and obtains the merged model. Second, the attacker inputs direct PII-related queries to the merged model to extract targeted PII. Extensive experiments demonstrate that \texttt{Merger-as-a-Stealer} successfully executes attacks against various LLMs and model merging methods across diverse settings, highlighting the effectiveness of the proposed framework. Given that this attack enables character-level extraction for targeted PII without requiring any additional knowledge from the attacker, we stress the necessity for improved model alignment and more robust defense mechanisms to mitigate such threats.

Figures (3)

• Figure 1: Overview of Merger-as-a-Stealer. The left side illustrates the fine-tuning processes of the victim model and the attack model, resulting in an aligned model and a malicious model, respectively. The right side shows the degradation of the victim model's security awareness for PII-related queries before and after model merging. The merged model outputs the victim user's precise home address in response to the attacker's direct query, instead of rejecting such simple PII-related queries before model merging.
• Figure 2: Examples in shadow dataset. The instruction is a direct PII-related query. The output only includes an affirmative response prefix to instruction.
• Figure 3: Results (Exact / Mem / LCSp) of our attack on five PII types from LeakPII against Qwen-14B.