BadMerging: Backdoor Attacks Against Model Merging

TL;DR

This paper addresses the security risks of Model Merging (MM) by introducing BadMerging, a backdoor attack designed to survive the interpolation over merging coefficients. BadMerging employs a two-stage process and a novel feature-interpolation-based loss (FI loss) to maintain backdoor effectiveness across on-task and off-task settings, even when the adversary provides only a single backdoored task-specific model. Extensive experiments across multiple MM algorithms, datasets, and model backbones demonstrate that BadMerging outperforms existing backdoor attacks while preserving model utility, and standard defenses fail to detect or mitigate it. The work highlights a critical security gap in MM and urges the development of MM-specific defenses and safer model-sharing practices.

Abstract

Fine-tuning pre-trained models for downstream tasks has led to a proliferation of open-sourced task-specific models. Recently, Model Merging (MM) has emerged as an effective approach to facilitate knowledge transfer among these independently fine-tuned models. MM directly combines multiple fine-tuned task-specific models into a merged model without additional training, and the resulting model shows enhanced capabilities in multiple tasks. Although MM provides great utility, it may come with security risks because an adversary can exploit MM to affect multiple downstream tasks. However, the security risks of MM have barely been studied. In this paper, we first find that MM, as a new learning paradigm, introduces unique challenges for existing backdoor attacks due to the merging process. To address these challenges, we introduce BadMerging, the first backdoor attack specifically designed for MM. Notably, BadMerging allows an adversary to compromise the entire merged model by contributing as few as one backdoored task-specific model. BadMerging comprises a two-stage attack mechanism and a novel feature-interpolation-based loss to enhance the robustness of embedded backdoors against the changes of different merging parameters. Considering that a merged model may incorporate tasks from different domains, BadMerging can jointly compromise the tasks provided by the adversary (on-task attack) and other contributors (off-task attack) and solve the corresponding unique challenges with novel attack designs. Extensive experiments show that BadMerging achieves remarkable attacks against various MM algorithms. Our ablation study demonstrates that the proposed attack designs can progressively contribute to the attack performance. Finally, we show that prior defense mechanisms fail to defend against our attacks, highlighting the need for more advanced defense.

Figures (11)

• Figure 1: Fine-tuning and merging task-specific models.
• Figure 2: An illustration of BadMerging. The adversary provides a backdoored CIFAR100 model. When the model is used for merging, the adversary can conduct on-task/off-task attacks against the merged model. (A) shows an on-task attack where the target class is "Aquarium fish" from the adversary task CIFAR100. (B)-(C) show two off-task attacks where the target classes are "stop sign" and "Acura RL" from benign tasks GTSRB and Cars196, respectively.
• Figure 3: In each figure, we plot the features of triggered images extracted by the visual encoder of backdoored merged models with different $\lambda_{{\text{adv}}}$ (i.e., in different colors). Features of triggered images form a compact cluster (yellow region) when $\lambda_{{\text{adv}}}=1$. Moreover, we observe interpolation property among the features extracted under different $\lambda_{{\text{adv}}}$: As the $\lambda_{{\text{adv}}}$ increases, the feature of a triggered image changes, closely following the red arrow.
• Figure 4: Each figure shows features of a triggered image under different $\lambda_{{\text{adv}}}$. Existing attacks fail because they only make triggered images predicted as the target class when $\lambda_{{\text{adv}}}$ is large. BadMerging uses the universal trigger and FI loss to robustify triggered images against various $\lambda_{{\text{adv}}}$.
• Figure 5: The pipeline of adversarial data augmentation.