Protecting Model Adaptation from Trojans in the Unlabeled Data

TL;DR

This work exposes the vulnerability of model adaptation to backdoor attacks launched via unlabeled target data, showing that carefully designed triggers and poisoning strategies can achieve high attack success without degrading clean performance. It proposes DiffAdapt, a plug-and-play defense that down-weights potentially poisoned unlabeled samples by estimating sensitivity to perturbations on a risk model and aggregating weights by pseudo-labels, enabling secure adaptation without altering underlying algorithms. Empirical results across Office, OfficeHome, and miniDomainNet demonstrate that DiffAdapt substantially reduces attack success rates while preserving target-domain accuracy, outperforming pruning-based baselines. The study highlights the need for security-conscious design in transfer learning with unlabeled data and provides practical guidance for defending source-free adaptation pipelines in real-world deployments.

Abstract

Model adaptation tackles the distribution shift problem with a pre-trained model instead of raw data, which has become a popular paradigm due to its great privacy protection. Existing methods always assume adapting to a clean target domain, overlooking the security risks of unlabeled samples. This paper for the first time explores the potential trojan attacks on model adaptation launched by well-designed poisoning target data. Concretely, we provide two trigger patterns with two poisoning strategies for different prior knowledge owned by attackers. These attacks achieve a high success rate while maintaining the normal performance on clean samples in the test stage. To defend against such backdoor injection, we propose a plug-and-play method named DiffAdapt, which can be seamlessly integrated with existing adaptation algorithms. Experiments across commonly used benchmarks and adaptation methods demonstrate the effectiveness of DiffAdapt. We hope this work will shed light on the safety of transfer learning with unlabeled data.

Figures (4)

• Figure 1: Backdoor attack and defense on model adaptation. With well-poisoned unlabeled data from malicious providers, target users suffer from the risks of backdoor injection. We propose DiffAdapt, a defense method against backdoor injection without sacrificing clean performance.
• Figure 2: The framework of defense method DiffAdapt. We train a potentially risky target model and obtain the distance between the output prediction of the original and perturbed version for every unlabeled data. The sample weight is averaged by the distance of all samples with the same pseudo label and a new secure target model is trained with the sample weight.
• Figure 3: ACC and ASR curve of backdoor attack and DiffAdapt on C$\to$P from miniDomainNetpeng2019moment.
• Figure 4: ACC (%) and ASR (%) of backdoor attacks under different poisoning rates for model adaptation.