Cross-Entropy Attacks to Language Models via Rare Event Simulation

TL;DR

This work tackles the challenge of black-box textual adversarial attacks by introducing Cross-Entropy Attacks (CEA), which recast adversarial generation as rare-event optimization guided by cross-entropy. It defines task-specific objective functions for both classifiers (soft-label and hard-label) and neural machine translation, and employs sememe-informed word substitutions to preserve semantics. Through extensive experiments on classification, translation, and LLM scenarios, CEA achieves superior attack effectiveness, imperceptibility, and linguistic quality compared with strong baselines, and demonstrates robustness under defenses and transferability conditions. The results highlight CEA as a principled, adaptable framework for evaluating and enhancing NLP robustness, with implications for adversarial training and defense design.

Abstract

Black-box textual adversarial attacks are challenging due to the lack of model information and the discrete, non-differentiable nature of text. Existing methods often lack versatility for attacking different models, suffer from limited attacking performance due to the inefficient optimization with word saliency ranking, and frequently sacrifice semantic integrity to achieve better attack outcomes. This paper introduces a novel approach to textual adversarial attacks, which we call Cross-Entropy Attacks (CEA), that uses Cross-Entropy optimization to address the above issues. Our CEA approach defines adversarial objectives for both soft-label and hard-label settings and employs CE optimization to identify optimal replacements. Through extensive experiments on document classification and language translation problems, we demonstrate that our attack method excels in terms of attacking performance, imperceptibility, and sentence quality.

Figures (6)

• Figure 1: SAR comparison of CEA, PSO+Obj, and RJS+Obj in Soft-label and Hard-label attacks on the AG News dataset using a TextCNN model.
• Figure 2: Ablation study results of CEA hyperparameters on SAR and BD metrics across classification and NMT tasks.
• Figure 3: Performance comparison of adversarial attacks against BERT and T5 using two defense mechanisms, FGWS and RanMASK, across IMDB, SST2 and WMT T1 datasets.
• Figure 4: Performance of transfer hard-label attacks to victim models (BERT and TextCNN) on AG News. A lower accuracy of the victim models indicates a higher transfer ability (i.e., the lower, the better).
• Figure 5: Performance of adversarially trained BERT with SST2 after joining different percentage of adversarial examples into the training set.