UOR: Universal Backdoor Attacks on Pre-trained Language Models

TL;DR

This work introduces UOR, a universal backdoor attack on pre-trained language models that is task-agnostic and transferable across downstream tasks. It replaces manually designed trigger-output representations with poisoned supervised contrastive learning (PSCL) to learn Uniform and Universal Output Representations (UORs) and uses gradient search to optimize trigger words, yielding stronger and more universal backdoors. The approach includes a two-term training loss $\,\mathcal{L}_B=\mathcal{L}_p+\lambda\mathcal{L}_c$ that couples PSCL with feature-alignment to preserve clean accuracy, and evaluates across multiple PLMs (e.g., BERT, XLNet, BART) and usage paradigms (fine-tuning, prompt-tuning, p-tuning) on diverse tasks, including classification, multiple-choice, and NER. Extensive experiments demonstrate superior universality and robustness against several defenses, underscoring a significant security threat and informing defense strategies for PLM sharing platforms.

Abstract

Backdoors implanted in pre-trained language models (PLMs) can be transferred to various downstream tasks, which exposes a severe security threat. However, most existing backdoor attacks against PLMs are un-targeted and task-specific. Few targeted and task-agnostic methods use manually pre-defined triggers and output representations, which prevent the attacks from being more effective and general. In this paper, we first summarize the requirements that a more threatening backdoor attack against PLMs should satisfy, and then propose a new backdoor attack method called UOR, which breaks the bottleneck of the previous approach by turning manual selection into automatic optimization. Specifically, we define poisoned supervised contrastive learning which can automatically learn the more uniform and universal output representations of triggers for various PLMs. Moreover, we use gradient search to select appropriate trigger words which can be adaptive to different PLMs and vocabularies. Experiments show that our method can achieve better attack performance on various text classification tasks compared to manual methods. Further, we tested our method on PLMs with different architectures, different usage paradigms, and more difficult tasks, which demonstrated the universality of our method.

Figures (5)

• Figure 1: An illustration of task-agnostic backdoor attacks against PLMs by manual methods and our methods.
• Figure 2: The pipeline and threat model of UOR.
• Figure 3: The framework of UOR, where the gradient search module selects trigger words and the poisoned supervised contrastive learning module injects backdoors into the PLMs. Backdoors implanted in PLMs will be migrated and inherited to downstream models after downstream tuning.
• Figure 4: Results of Re-init and Fine-Pruning defense. Re-initialization is performed for the last layer (LL), the pooler layer (PL) and both of them (LL+PL). Fine-Pruning is performed with the prune rate from 0.2 to 0.99.
• Figure 5: Visualization of dimension-reduced UORs on clean and backdoored BERTs, and clean and backdoored SST-2 downstream models.