Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Adversarial Training against Textual Adversarial Attacks

Yichen Yang, Xin Liu, Kun He

TL;DR

This work addresses NLP adversarial robustness under synonym-unaware conditions by proposing Fast Adversarial Training (FAT), an embedding-space defense that uses single-step gradient ascent with history-informed perturbation initialization to accelerate training. The approach is supported by empirical findings that single-step perturbations closely approximate multi-step ones in the embedding space and that perturbations exhibit directional consistency across epochs. FAT and its interval variant FAT-$I$ demonstrate superior robustness on BERT across IMDB, AGNEWS, and QNLI against both word- and character-level attacks, while maintaining competitive clean accuracy and offering substantial training-time savings. The results indicate a practical, language-agnostic defense that does not rely on attacker synonym information, highlighting FAT as a strong baseline for synonym-unaware adversarial training in NLP.

Abstract

Many adversarial defense methods have been proposed to enhance the adversarial robustness of natural language processing models. However, most of them introduce additional pre-set linguistic knowledge and assume that the synonym candidates used by attackers are accessible, which is an ideal assumption. We delve into adversarial training in the embedding space and propose a Fast Adversarial Training (FAT) method to improve the model robustness in the synonym-unaware scenario from the perspective of single-step perturbation generation and perturbation initialization. Based on the observation that the adversarial perturbations crafted by single-step and multi-step gradient ascent are similar, FAT uses single-step gradient ascent to craft adversarial examples in the embedding space to expedite the training process. Based on the observation that the perturbations generated on the identical training sample in successive epochs are similar, FAT fully utilizes historical information when initializing the perturbation. Extensive experiments demonstrate that FAT significantly boosts the robustness of BERT models in the synonym-unaware scenario, and outperforms the defense baselines under various attacks with character-level and word-level modifications.

Fast Adversarial Training against Textual Adversarial Attacks

TL;DR

This work addresses NLP adversarial robustness under synonym-unaware conditions by proposing Fast Adversarial Training (FAT), an embedding-space defense that uses single-step gradient ascent with history-informed perturbation initialization to accelerate training. The approach is supported by empirical findings that single-step perturbations closely approximate multi-step ones in the embedding space and that perturbations exhibit directional consistency across epochs. FAT and its interval variant FAT- demonstrate superior robustness on BERT across IMDB, AGNEWS, and QNLI against both word- and character-level attacks, while maintaining competitive clean accuracy and offering substantial training-time savings. The results indicate a practical, language-agnostic defense that does not rely on attacker synonym information, highlighting FAT as a strong baseline for synonym-unaware adversarial training in NLP.

Abstract

Many adversarial defense methods have been proposed to enhance the adversarial robustness of natural language processing models. However, most of them introduce additional pre-set linguistic knowledge and assume that the synonym candidates used by attackers are accessible, which is an ideal assumption. We delve into adversarial training in the embedding space and propose a Fast Adversarial Training (FAT) method to improve the model robustness in the synonym-unaware scenario from the perspective of single-step perturbation generation and perturbation initialization. Based on the observation that the adversarial perturbations crafted by single-step and multi-step gradient ascent are similar, FAT uses single-step gradient ascent to craft adversarial examples in the embedding space to expedite the training process. Based on the observation that the perturbations generated on the identical training sample in successive epochs are similar, FAT fully utilizes historical information when initializing the perturbation. Extensive experiments demonstrate that FAT significantly boosts the robustness of BERT models in the synonym-unaware scenario, and outperforms the defense baselines under various attacks with character-level and word-level modifications.
Paper Structure (22 sections, 7 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 22 sections, 7 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Rethinking Adversarial Training
  5. Fast Adversarial Training
  6. Single-Step Perturbation Generation
  7. Perturbation Initialization
  8. Variants of FAT
  9. Experiments
  10. Experimental Setup
  11. Tasks and Models
  12. Attack Methods
  13. Defense Baselines
  14. Training Details
  15. Main Results
  16. ...and 7 more sections

Figures (4)

  • Figure 1: The direction similarity between perturbations generated by single-step and multi-step gradient ascent in the embedding space.
  • Figure 2: The direction similarity between perturbations generated by last and current epoch in the embedding space.
  • Figure 3: The impact of hyper-parameter $\epsilon$ on the performance of FAT across the three datasets.
  • Figure 4: The impact of training epochs on the performance of FAT across the three datasets.