VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models

TL;DR

The paper reveals a security risk in instruction-tuned autoregressive visual-language models by introducing VL-Trojan, a multimodal backdoor that learns an image trigger via isolated clustering and a text trigger via iterative search, effective even when the visual encoder is frozen. It demonstrates high attack success rates with only tens of poisoned samples and shows robust transferability across model scales and tasks, including black-box settings when combining image and text triggers. Key contributions include identifying constraints of frozen encoders, designing a novel trigger learning strategy, and providing extensive experiments that outperform baselines. The work highlights practical security implications for multimodal VLMs and motivates defenses against backdoor manipulation in instruction-tuned systems.

Abstract

Autoregressive Visual Language Models (VLMs) showcase impressive few-shot learning capabilities in a multimodal context. Recently, multimodal instruction tuning has been proposed to further enhance instruction-following abilities. However, we uncover the potential threat posed by backdoor attacks on autoregressive VLMs during instruction tuning. Adversaries can implant a backdoor by injecting poisoned samples with triggers embedded in instructions or images, enabling malicious manipulation of the victim model's predictions with predefined triggers. Nevertheless, the frozen visual encoder in autoregressive VLMs imposes constraints on the learning of conventional image triggers. Additionally, adversaries may encounter restrictions in accessing the parameters and architectures of the victim model. To address these challenges, we propose a multimodal instruction backdoor attack, namely VL-Trojan. Our approach facilitates image trigger learning through an isolating and clustering strategy and enhance black-box-attack efficacy via an iterative character-level text trigger generation method. Our attack successfully induces target outputs during inference, significantly surpassing baselines (+62.52\%) in ASR. Moreover, it demonstrates robustness across various model scales and few-shot in-context reasoning scenarios.

Figures (8)

• Figure 1: Overall framework of our multimodal instruction backdoor attack on autoregressive VLMs.
• Figure 2: Motivation for our proposed isolated clustering approach. In the context where the visual encoder is frozen during training, conventional backdoor triggers lead to a high overlap between features of backdoor images and clean images, as illustrated in (b). In contrast, these features are effectively separated into two clusters when the visual encoder is trainable, as illustrated in (a). Motivated by this observation, we aim to propose a backdoor trigger capable of manipulating the features of poisoned images so that their distribution aligns with those in a trainable visual encoder.
• Figure 3: Ablation study on the poisoning rate. As the poisoning rate increases, a decline in accuracy on clean samples is observed, accompanied by an increase in ASR on poisoned samples. Our approach maintains a high ASR even at low poisoning rates, in contrast to other baselines that hinge on high poisoning rates for effective backdoor attacks.
• Figure 4: Visualization of the features of clean and poisoned images from the vision encoder (ViT-L/14) of victim model. Poisoned images are embedded with backdoor triggers crafted using different visual encoders.
• Figure 5: Ablation on model scales. We conduct experiments using OpenFlamingo models of scale 3B, 4B, and 9B, respectively.