Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attacks

Oscar Chew, Po-Yi Lu, Jayden Lin, Hsuan-Tien Lin

TL;DR

Experiments show that textual perturbations are effective in defending against state-of-the-art backdoor attacks with minimal sacrifice to generation quality, and the efficacy of textual perturbation is analyzed from two angles: text embedding space and cross-attention maps.

Abstract

Text-to-image diffusion models have been widely adopted in real-world applications due to their ability to generate realistic images from textual descriptions. However, recent studies have shown that these methods are vulnerable to backdoor attacks. Despite the significant threat posed by backdoor attacks on text-to-image diffusion models, countermeasures remain under-explored. In this paper, we address this research gap by demonstrating that state-of-the-art backdoor attacks against text-to-image diffusion models can be effectively mitigated by a surprisingly simple defense strategy - textual perturbation. Experiments show that textual perturbations are effective in defending against state-of-the-art backdoor attacks with minimal sacrifice to generation quality. We analyze the efficacy of textual perturbation from two angles: text embedding space and cross-attention maps. They further explain how backdoor attacks have compromised text-to-image diffusion models, providing insights for studying future attack and defense strategies. Our code is available at https://github.com/oscarchew/t2i-backdoor-defense.

Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attacks

TL;DR

Experiments show that textual perturbations are effective in defending against state-of-the-art backdoor attacks with minimal sacrifice to generation quality, and the efficacy of textual perturbation is analyzed from two angles: text embedding space and cross-attention maps.

Abstract

Text-to-image diffusion models have been widely adopted in real-world applications due to their ability to generate realistic images from textual descriptions. However, recent studies have shown that these methods are vulnerable to backdoor attacks. Despite the significant threat posed by backdoor attacks on text-to-image diffusion models, countermeasures remain under-explored. In this paper, we address this research gap by demonstrating that state-of-the-art backdoor attacks against text-to-image diffusion models can be effectively mitigated by a surprisingly simple defense strategy - textual perturbation. Experiments show that textual perturbations are effective in defending against state-of-the-art backdoor attacks with minimal sacrifice to generation quality. We analyze the efficacy of textual perturbation from two angles: text embedding space and cross-attention maps. They further explain how backdoor attacks have compromised text-to-image diffusion models, providing insights for studying future attack and defense strategies. Our code is available at https://github.com/oscarchew/t2i-backdoor-defense.
Paper Structure (26 sections, 3 figures, 6 tables)

This paper contains 26 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Image Diffusion Model
  4. Backdoor Attack against Text-to-Image Diffusion Models
  5. Backdoor Defense for Diffusion Models
  6. Textual Perturbation as a Remedy
  7. Word-level Perturbation
  8. Character-level Perturbation
  9. Experiments
  10. Experiment Setup
  11. Models
  12. Datasets
  13. Metrics
  14. Qualitative Results
  15. Quantitative Results
  16. ...and 11 more sections

Figures (3)

  • Figure 1: t-SNE projection of the text embedding space before and after applying Textual Inversion attack. The trigger token (beautiful car), target token (chow chow), and perturbed trigger (e.g. beautiful automobile) are highlighted in blue, red and green.
  • Figure 2: Framework Designed to Defend Against Backdoor Attacks
  • Figure 3: t-SNE projection of the text embedding space before and after applying Rickrolling attack. The trigger token (U+0B20), target token (A lightning strike), and perturbed trigger (e.g. o) are highlighted in blue, red and green respectively.