PETA: Parameter-Efficient Trojan Attacks
Lauren Hong, Ting Wang
TL;DR
The paper addresses security risks in parameter-efficient fine-tuning (PEFT) by introducing PETA, a backdoor attack that uses bilevel optimization to embed a backdoor into a pre-trained language model while simulating downstream PEFT to ensure persistence after fine-tuning. The attack poisons data partitions and optimizes an outer backdoor objective L_atk together with a lower-level PEFT objective L_peft, producing a backdoored model f(·; θ*) that remains effective when the victim later applies PEFT with their own data and triggers. Across multiple tasks, triggers, and attacker-knowledge settings, PETA achieves high attack success while maintaining clean accuracy, outperforming baselines, and proving robust to domain shifts and unknown PEFT techniques. The findings raise important security concerns for PEFT deployments and underscore the need for defenses against such bilevel-backdoor strategies.
Abstract
Parameter-efficient fine-tuning (PEFT) enables efficient adaptation of pre-trained language models (PLMs) to specific tasks. By tuning only a minimal set of (extra) parameters, PEFT achieves performance that is comparable to standard fine-tuning. However, despite its prevalent use, the security implications of PEFT remain largely unexplored. In this paper, we take the initial steps and present PETA, a novel trojan attack that compromises the weights of PLMs by accounting for downstream adaptation through bilevel optimization: the upper-level objective embeds the backdoor into a model while the lower-level objective simulates PEFT to both retain the PLM's task-specific performance and ensure that the backdoor persists after fine-tuning. With extensive evaluation across a variety of downstream tasks and trigger designs, we demonstrate PETA's effectiveness in terms of both attack success rate and clean accuracy, even when the attacker does not have full knowledge of the victim user's training process.