Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enabling Heterogeneous Adversarial Transferability via Feature Permutation Attacks

Tao Wu, Tie Luo

TL;DR

The paper tackles the challenge of adversarial transferability across heterogeneous architectures (CNNs, ViTs, and MLPs) by introducing Feature Permutation Attack (FPA), a zero-FLOP, parameter-free layer that permutes internal CNN feature maps to induce long-range dependencies. FPA comes in two strategies, FPA-R and FPA-N, and acts as a plug-and-play enhancement that can boost existing transfer-based attacks without altering their core mechanisms. Across 14 architectures on ImageNet, FPA substantially improves transferability, with notable gains for ViTs and MLPs and compatible gains when combined with other attacks, all while adding negligible computational overhead. The method provides a practical, architecture-agnostic approach to strengthening black-box attack effectiveness and offers insights into leveraging feature-level interactions to bridge architectural gaps.

Abstract

Adversarial attacks in black-box settings are highly practical, with transfer-based attacks being the most effective at generating adversarial examples (AEs) that transfer from surrogate models to unseen target models. However, their performance significantly degrades when transferring across heterogeneous architectures -- such as CNNs, MLPs, and Vision Transformers (ViTs) -- due to fundamental architectural differences. To address this, we propose Feature Permutation Attack (FPA), a zero-FLOP, parameter-free method that enhances adversarial transferability across diverse architectures. FPA introduces a novel feature permutation (FP) operation, which rearranges pixel values in selected feature maps to simulate long-range dependencies, effectively making CNNs behave more like ViTs and MLPs. This enhances feature diversity and improves transferability both across heterogeneous architectures and within homogeneous CNNs. Extensive evaluations on 14 state-of-the-art architectures show that FPA achieves maximum absolute gains in attack success rates of 7.68% on CNNs, 14.57% on ViTs, and 14.48% on MLPs, outperforming existing black-box attacks. Additionally, FPA is highly generalizable and can seamlessly integrate with other transfer-based attacks to further boost their performance. Our findings establish FPA as a robust, efficient, and computationally lightweight strategy for enhancing adversarial transferability across heterogeneous architectures.

Enabling Heterogeneous Adversarial Transferability via Feature Permutation Attacks

TL;DR

The paper tackles the challenge of adversarial transferability across heterogeneous architectures (CNNs, ViTs, and MLPs) by introducing Feature Permutation Attack (FPA), a zero-FLOP, parameter-free layer that permutes internal CNN feature maps to induce long-range dependencies. FPA comes in two strategies, FPA-R and FPA-N, and acts as a plug-and-play enhancement that can boost existing transfer-based attacks without altering their core mechanisms. Across 14 architectures on ImageNet, FPA substantially improves transferability, with notable gains for ViTs and MLPs and compatible gains when combined with other attacks, all while adding negligible computational overhead. The method provides a practical, architecture-agnostic approach to strengthening black-box attack effectiveness and offers insights into leveraging feature-level interactions to bridge architectural gaps.

Abstract

Adversarial attacks in black-box settings are highly practical, with transfer-based attacks being the most effective at generating adversarial examples (AEs) that transfer from surrogate models to unseen target models. However, their performance significantly degrades when transferring across heterogeneous architectures -- such as CNNs, MLPs, and Vision Transformers (ViTs) -- due to fundamental architectural differences. To address this, we propose Feature Permutation Attack (FPA), a zero-FLOP, parameter-free method that enhances adversarial transferability across diverse architectures. FPA introduces a novel feature permutation (FP) operation, which rearranges pixel values in selected feature maps to simulate long-range dependencies, effectively making CNNs behave more like ViTs and MLPs. This enhances feature diversity and improves transferability both across heterogeneous architectures and within homogeneous CNNs. Extensive evaluations on 14 state-of-the-art architectures show that FPA achieves maximum absolute gains in attack success rates of 7.68% on CNNs, 14.57% on ViTs, and 14.48% on MLPs, outperforming existing black-box attacks. Additionally, FPA is highly generalizable and can seamlessly integrate with other transfer-based attacks to further boost their performance. Our findings establish FPA as a robust, efficient, and computationally lightweight strategy for enhancing adversarial transferability across heterogeneous architectures.

Paper Structure

This paper contains 16 sections, 6 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transfer-based Attacks on CNNs
  4. Transfer-based Attacks on ViTs
  5. Method
  6. Feature Permutation Attack (FPA)
  7. Analysis of Why FPA Works
  8. Experiments
  9. Experimental Setup
  10. Experimental Results
  11. Ablation studies
  12. Ratio of Permuted Channels ($\gamma$)
  13. Probability of Permutation ($p$)
  14. Position of the FP Layer
  15. Wall Clock Runtime
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Adversarial transferability collapses across heterogeneous architectures. AEs generated on ResNet-50 transfers well to CNNs like DenseNet121 but fail drastically on ViTs and MLPs (tested with I-FGSM and MI-FGSM). Our proposed FPA-N significantly improves this by increasing success rates by $\sim$20 percentage points.
  • Figure 2: Illustration of the two proposed FP strategies: (a) FPA-R enables feature interactions in broader ranges, while (b) FPA-N maintains spatial coherence through local exchanges.
  • Figure 3: Feature maps after applying the two FP strategies with probability $p$.
  • Figure 4: Ablation study on key hyper-parameters of FPA-R and FPA-N: (left) ratio of permuted channels $\gamma$, (center) probability of permutation $p$, and (right) position of the inserted FPA layer $l$.