MTAttack: Multi-Target Backdoor Attacks against Large Vision-Language Models

TL;DR

This work introduces MTAttack, the first framework for multi-target backdoor attacks against Large Vision-Language Models. It tackles inter-trigger interference by jointly optimizing multiple visual triggers under two constraints—Proxy Space Partitioning ($\mathcal{L}_{PSP}$) to separate proxy target concepts and Trigger Prototype Anchoring ($\mathcal{L}_{TPA}$) to cluster poisoned samples around learnable prototypes—thereby establishing accurate one-to-one trigger-target mappings during backdoor implanting. Through extensive experiments on multiple state-of-the-art LVLMs and standard captioning benchmarks, MTAttack consistently outperforms single-target baselines and existing multi-target methods in attack success rate while maintaining clean-output quality, demonstrating strong generalization across datasets and robustness against defense techniques. The results underscore a pressing security risk in LVLMs and highlight the need for defense strategies that can detect and mitigate multi-target backdoors in multimodal, autoregressive settings. Code is available at https://github.com/mala-lab/MTAttack.

Abstract

Recent advances in Large Visual Language Models (LVLMs) have demonstrated impressive performance across various vision-language tasks by leveraging large-scale image-text pretraining and instruction tuning. However, the security vulnerabilities of LVLMs have become increasingly concerning, particularly their susceptibility to backdoor attacks. Existing backdoor attacks focus on single-target attacks, i.e., targeting a single malicious output associated with a specific trigger. In this work, we uncover multi-target backdoor attacks, where multiple independent triggers corresponding to different attack targets are added in a single pass of training, posing a greater threat to LVLMs in real-world applications. Executing such attacks in LVLMs is challenging since there can be many incorrect trigger-target mappings due to severe feature interference among different triggers. To address this challenge, we propose MTAttack, the first multi-target backdoor attack framework for enforcing accurate multiple trigger-target mappings in LVLMs. The core of MTAttack is a novel optimization method with two constraints, namely Proxy Space Partitioning constraint and Trigger Prototype Anchoring constraint. It jointly optimizes multiple triggers in the latent space, with each trigger independently mapping clean images to a unique proxy class while at the same time guaranteeing their separability. Experiments on popular benchmarks demonstrate a high success rate of MTAttack for multi-target attacks, substantially outperforming existing attack methods. Furthermore, our attack exhibits strong generalizability across datasets and robustness against backdoor defense strategies. These findings highlight the vulnerability of LVLMs to multi-target backdoor attacks and underscore the urgent need for mitigating such threats. Code is available at https://github.com/mala-lab/MTAttack.

Figures (6)

• Figure 1: Illustration of multi-target backdoor attacks. The goal is to poison a victim LVLM such that different triggers are bound to multiple attack targets after a single fine-tuning pass. The LVLM then generates (incorrrect) target text outputs when queried by images with any of the triggers.
• Figure 2: Top: t-SNE visualization of features extracted by the vision encoder of a victim LVLM for clean and poisoned images. Bottom: Comparison of attack success rates of ReFool, VL-Trojan, and MTAttack in the 4-target setting. "Success" means the model correctly classifies the target as defined by the trigger, "Confusion" means the model yields an attack target but mismatches with the designated trigger, and "Failed" indicates no backdoor target is activated.
• Figure 3: Overview of MTAttack. It first learns multiple visual triggers with the Proxy Space Partitioning (PSP) and Trigger Prototype Anchoring (TPA) constraints. PSP maximizes the separation between the proxy classes of the triggers, while TPA ensures the poisoned samples align closely with the learnable prototype of their proxy class. To implant the backdoor with the triggers, MTAttack then establishes one-to-one mappings between different triggers and their corresponding text attack targets.
• Figure 4: t-SNE visualization of features extracted by the vision encoder for clean images, poisoned images after applying the trigger, and original images of the target concept.
• Figure 5: Attack effectiveness under further fine-tuning with benign data on Flickr-30K.