BadApex: Backdoor Attack Based on Adaptive Optimization Mechanism of Black-box Large Language Models

TL;DR

BadApex introduces a two-stage backdoor framework that leverages a dual-agent Adaptive Optimization Mechanism to refine a prompt guiding black-box LLMs in generating stealthy, high-quality poisoned texts. The methodology couples prompt refinement (AOM) with a poisoned-text generator (PTGM) across multiple LLMs, achieving high attack efficacy ($ASR$ ≈ 98%) while preserving clean accuracy and maintaining stealth under defenses. Extensive experiments on OLID, SST2, and AGNews show superior adaptability across LLMs, improved semantic fluency and coherence, and robust attack performance even when defenses like ONION and TextGuard are applied. This work highlights the need for stronger defenses against adaptive, LLM-guided backdoors and informs security researchers about potential avenues of evasion and resilience, with practical implications for evaluating and fortifying NLP systems against covert manipulations.

Abstract

Previous insertion-based and paraphrase-based backdoors have achieved great success in attack efficacy, but they ignore the text quality and semantic consistency between poisoned and clean texts. Although recent studies introduce LLMs to generate poisoned texts and improve the stealthiness, semantic consistency, and text quality, their hand-crafted prompts rely on expert experiences, facing significant challenges in prompt adaptability and attack performance after defenses. In this paper, we propose a novel backdoor attack based on adaptive optimization mechanism of black-box large language models (BadApex), which leverages a black-box LLM to generate poisoned text through a refined prompt. Specifically, an Adaptive Optimization Mechanism is designed to refine an initial prompt iteratively using the generation and modification agents. The generation agent generates the poisoned text based on the initial prompt. Then the modification agent evaluates the quality of the poisoned text and refines a new prompt. After several iterations of the above process, the refined prompt is used to generate poisoned texts through LLMs. We conduct extensive experiments on three dataset with six backdoor attacks and two defenses. Extensive experimental results demonstrate that BadApex significantly outperforms state-of-the-art attacks. It improves prompt adaptability, semantic consistency, and text quality. Furthermore, when two defense methods are applied, the average attack success rate (ASR) still up to 96.75%.

Figures (6)

• Figure 1: (a) Previous insertion-based methods. (b) Previous paraphrase-based methods ignore semantic consistency and text quality of poisoned texts. (c) Our proposed method generates stealthy and efficiency poisoned texts via LLMs based on a refined prompt.
• Figure 2: Framework of BadApex. (a) Adaptive Optimization Mechanism refines a new prompt from hand-crafted prompt iteratively using generation agent $A_g$ and modification agent $A_m$ after $n$ iterations. (b) Poisoned Text Generation Module generates poisoned data using one of alterative LLMs based on refined prompt $P_n$.
• Figure 3: Case study of using GPT-4 as the Generation Agent $A_g$. Current prompt is hand-crafted $P_o$, and poisoned text is best rewritten text $x^*$.
• Figure 4: Case study of using GPT-4 as the Modification Agent $A_m$. $P_1$ is the refined version of $P_o$.
• Figure 5: Average PPL$_{1}$ and PPL$_{2}$ of 1,000 poisoned texts from different backdoor attacks.