Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Spatial Alignment Framework for Highly Transferable Transformation-Based Attacks on Spatially Structured Tasks

Jiaming Liang, Chi-Man Pun

Abstract

Transformation-based adversarial attacks (TAAs) demonstrate strong transferability when deceiving classification models. However, existing TAAs often perform unsatisfactorily or even fail when applied to structured tasks such as semantic segmentation and object detection. Encouragingly, recent studies that categorize transformations into non-spatial and spatial transformations inspire us to address this challenge. We find that for non-structured tasks, labels are spatially non-structured, and thus TAAs are not required to adjust labels when applying spatial transformations. In contrast, for structured tasks, labels are spatially structured, and failing to transform labels synchronously with inputs can cause spatial misalignment and yield erroneous gradients. To address these issues, we propose a novel unified Spatial Alignment Framework (SAF) for highly transferable TAAs on spatially structured tasks, where the TAAs spatially transform labels synchronously with the input using the proposed Spatial Alignment (SA) algorithm. Extensive experiments demonstrate the crucial role of our SAF for TAAs on structured tasks. Specifically, in non-targeted attacks, our SAF degrades the average mIoU on Cityscapes from 24.50 to 11.34, and on Kvasir-SEG from 49.91 to 31.80, while reducing the average mAP of COCO from 17.89 to 5.25.

A Unified Spatial Alignment Framework for Highly Transferable Transformation-Based Attacks on Spatially Structured Tasks

Abstract

Transformation-based adversarial attacks (TAAs) demonstrate strong transferability when deceiving classification models. However, existing TAAs often perform unsatisfactorily or even fail when applied to structured tasks such as semantic segmentation and object detection. Encouragingly, recent studies that categorize transformations into non-spatial and spatial transformations inspire us to address this challenge. We find that for non-structured tasks, labels are spatially non-structured, and thus TAAs are not required to adjust labels when applying spatial transformations. In contrast, for structured tasks, labels are spatially structured, and failing to transform labels synchronously with inputs can cause spatial misalignment and yield erroneous gradients. To address these issues, we propose a novel unified Spatial Alignment Framework (SAF) for highly transferable TAAs on spatially structured tasks, where the TAAs spatially transform labels synchronously with the input using the proposed Spatial Alignment (SA) algorithm. Extensive experiments demonstrate the crucial role of our SAF for TAAs on structured tasks. Specifically, in non-targeted attacks, our SAF degrades the average mIoU on Cityscapes from 24.50 to 11.34, and on Kvasir-SEG from 49.91 to 31.80, while reducing the average mAP of COCO from 17.89 to 5.25.

Paper Structure

This paper contains 17 sections, 8 equations, 6 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adversarial Attacks for Structured Tasks
  4. Transformation-based Adversarial Attacks
  5. Methodology
  6. Preliminaries
  7. Causes of TAAs Failure in Structured Tasks
  8. Spatial Alignment
  9. Experiments and Results
  10. Setup
  11. Non-Targeted Attacks on Semantic Segmentation
  12. Non-Targeted Attacks on Object Detection
  13. Targeted Attacks on Cityscapes
  14. Attacks under Input Preprocessing Defense
  15. Ablation Study on Iterations
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Performance gains from integrating spatial alignment framework into various TAAs across spatially structured tasks. Four groups of bars correspond to DEM, SIA, BSR, and I-C, respectively. Surrogate models are indicated in gray boxes.
  • Figure 2: Illustration of the proposed Spatial Alignment Framework (SAF).
  • Figure 3: Visualized non-targeted results of diverse transformation-based attacks with and without spatial alignment.
  • Figure 4: Visualized targeted results of diverse transformation-based attacks with and without spatial alignment.
  • Figure 5: Ablation studies on the number of iterations $L$.
  • ...and 1 more figures