Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BadCM: Invisible Backdoor Attack Against Cross-Modal Learning

Zheng Zhang, Xu Yuan, Lei Zhu, Jingkuan Song, Liqiang Nie

TL;DR

This paper introduces a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor and proposes a generalized invisible backdoor framework against cross-modal learning (BadCM), the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework.

Abstract

Despite remarkable successes in unimodal learning tasks, backdoor attacks against cross-modal learning are still underexplored due to the limited generalization and inferior stealthiness when involving multiple modalities. Notably, since works in this area mainly inherit ideas from unimodal visual attacks, they struggle with dealing with diverse cross-modal attack circumstances and manipulating imperceptible trigger samples, which hinders their practicability in real-world applications. In this paper, we introduce a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor and propose a generalized invisible backdoor framework against cross-modal learning (BadCM). Specifically, a cross-modal mining scheme is developed to capture the modality-invariant components as target poisoning areas, where well-designed trigger patterns injected into these regions can be efficiently recognized by the victim models. This strategy is adapted to different image-text cross-modal models, making our framework available to various attack scenarios. Furthermore, for generating poisoned samples of high stealthiness, we conceive modality-specific generators for visual and linguistic modalities that facilitate hiding explicit trigger patterns in modality-invariant regions. To the best of our knowledge, BadCM is the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework. Comprehensive experimental evaluations on two typical applications, i.e., cross-modal retrieval and VQA, demonstrate the effectiveness and generalization of our method under multiple kinds of attack scenarios. Moreover, we show that BadCM can robustly evade existing backdoor defenses. Our code is available at https://github.com/xandery-geek/BadCM.

BadCM: Invisible Backdoor Attack Against Cross-Modal Learning

TL;DR

This paper introduces a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor and proposes a generalized invisible backdoor framework against cross-modal learning (BadCM), the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework.

Abstract

Despite remarkable successes in unimodal learning tasks, backdoor attacks against cross-modal learning are still underexplored due to the limited generalization and inferior stealthiness when involving multiple modalities. Notably, since works in this area mainly inherit ideas from unimodal visual attacks, they struggle with dealing with diverse cross-modal attack circumstances and manipulating imperceptible trigger samples, which hinders their practicability in real-world applications. In this paper, we introduce a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor and propose a generalized invisible backdoor framework against cross-modal learning (BadCM). Specifically, a cross-modal mining scheme is developed to capture the modality-invariant components as target poisoning areas, where well-designed trigger patterns injected into these regions can be efficiently recognized by the victim models. This strategy is adapted to different image-text cross-modal models, making our framework available to various attack scenarios. Furthermore, for generating poisoned samples of high stealthiness, we conceive modality-specific generators for visual and linguistic modalities that facilitate hiding explicit trigger patterns in modality-invariant regions. To the best of our knowledge, BadCM is the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework. Comprehensive experimental evaluations on two typical applications, i.e., cross-modal retrieval and VQA, demonstrate the effectiveness and generalization of our method under multiple kinds of attack scenarios. Moreover, we show that BadCM can robustly evade existing backdoor defenses. Our code is available at https://github.com/xandery-geek/BadCM.
Paper Structure (16 sections, 9 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 16 sections, 9 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Backdoor Attack
  4. Backdoor Attack on NLP
  5. Cross-Modal Learning
  6. Preliminaries
  7. Backdoor Framework for Cross-modal Learning
  8. Adversary’s Capacities
  9. Problem Definition of Cross-modal Retrieval
  10. Methodology
  11. Cross-modal Mining Scheme
  12. Visual Trigger Generation
  13. Textual Trigger Generation
  14. Experiment
  15. Experimental Settings
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Examples of the proposed bilateral backdoor attacks in the cross-modal retrieval, which include visual-to-linguistic (V2L) and linguistic-to-visual (L2V) attacks. The bilateral attacks aim to implant a backdoor from one modality and activate malicious behavior in the other one, which are complementary to dual-key attackswalmer2022dual.
  • Figure 2: The unified framework of the proposed BadCM consists of a cross-modal mining scheme, a visual trigger generator, and a textual trigger generator. The cross-modal mining scheme seeks to align vision and language to extract modality-invariant factors within each modality. The visual trigger generator produces poisoned images by transforming visible trigger patterns into invisible perturbations on the modality-invariant factors. Similarly, the textual trigger generator is designed to generate poisoned text by synonym substitution strategy.
  • Figure 3: Visual examples of poisoned images generated by different attacks on the NUS-WIDE dataset. Below the poisoned images, we also provide the corresponding residual maps between the original images and the trigger images. For FIBA, FTrojan, and our BadCM, we show the residual maps with $5 \times$ difference.