Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attacks

Xinyu Zhang, Hanbin Hong, Yuan Hong, Peng Huang, Binghui Wang, Zhongjie Ba, Kui Ren

TL;DR

Text-CRS introduces a generalized certified robustness framework for NLP that covers four fundamental word-level attacks by modeling them as permutation and embedding transformations. By pairing operation-specific randomized smoothing (Staircase, Uniform, Gaussian, Bernoulli) with an enhanced training toolkit, it delivers provable robustness bounds and competitive empirical performance across LSTM and BERT on AG’s News, Amazon, and IMDB. The framework provides the first certified guarantees for word reordering, insertion, and deletion, and sets new benchmarks for certified accuracy and radius under unseen attacks. Practically, Text-CRS offers a scalable, architecture-agnostic approach to provable NLP robustness with broad applicability to real-world adversarial scenarios.

Abstract

The language models, especially the basic text classification models, have been shown to be susceptible to textual adversarial attacks such as synonym substitution and word insertion attacks. To defend against such attacks, a growing body of research has been devoted to improving the model robustness. However, providing provable robustness guarantees instead of empirical robustness is still widely unexplored. In this paper, we propose Text-CRS, a generalized certified robustness framework for natural language processing (NLP) based on randomized smoothing. To our best knowledge, existing certified schemes for NLP can only certify the robustness against $\ell_0$ perturbations in synonym substitution attacks. Representing each word-level adversarial operation (i.e., synonym substitution, word reordering, insertion, and deletion) as a combination of permutation and embedding transformation, we propose novel smoothing theorems to derive robustness bounds in both permutation and embedding space against such adversarial operations. To further improve certified accuracy and radius, we consider the numerical relationships between discrete words and select proper noise distributions for the randomized smoothing. Finally, we conduct substantial experiments on multiple language models and datasets. Text-CRS can address all four different word-level adversarial operations and achieve a significant accuracy improvement. We also provide the first benchmark on certified accuracy and radius of four word-level operations, besides outperforming the state-of-the-art certification against synonym substitution attacks.

Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attacks

TL;DR

Text-CRS introduces a generalized certified robustness framework for NLP that covers four fundamental word-level attacks by modeling them as permutation and embedding transformations. By pairing operation-specific randomized smoothing (Staircase, Uniform, Gaussian, Bernoulli) with an enhanced training toolkit, it delivers provable robustness bounds and competitive empirical performance across LSTM and BERT on AG’s News, Amazon, and IMDB. The framework provides the first certified guarantees for word reordering, insertion, and deletion, and sets new benchmarks for certified accuracy and radius under unseen attacks. Practically, Text-CRS offers a scalable, architecture-agnostic approach to provable NLP robustness with broad applicability to real-world adversarial scenarios.

Abstract

The language models, especially the basic text classification models, have been shown to be susceptible to textual adversarial attacks such as synonym substitution and word insertion attacks. To defend against such attacks, a growing body of research has been devoted to improving the model robustness. However, providing provable robustness guarantees instead of empirical robustness is still widely unexplored. In this paper, we propose Text-CRS, a generalized certified robustness framework for natural language processing (NLP) based on randomized smoothing. To our best knowledge, existing certified schemes for NLP can only certify the robustness against perturbations in synonym substitution attacks. Representing each word-level adversarial operation (i.e., synonym substitution, word reordering, insertion, and deletion) as a combination of permutation and embedding transformation, we propose novel smoothing theorems to derive robustness bounds in both permutation and embedding space against such adversarial operations. To further improve certified accuracy and radius, we consider the numerical relationships between discrete words and select proper noise distributions for the randomized smoothing. Finally, we conduct substantial experiments on multiple language models and datasets. Text-CRS can address all four different word-level adversarial operations and achieve a significant accuracy improvement. We also provide the first benchmark on certified accuracy and radius of four word-level operations, besides outperforming the state-of-the-art certification against synonym substitution attacks.
Paper Structure (52 sections, 10 theorems, 52 equations, 14 figures, 10 tables, 2 algorithms)

This paper contains 52 sections, 10 theorems, 52 equations, 14 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Text Classification
  4. Adversarial Examples for Text Classification
  5. Threat Model
  6. Randomized Smoothing for Certified Defense
  7. The Text-CRS Framework
  8. New Challenges in Certified Defense Design
  9. Permutation and Embedding Transformation
  10. Notations
  11. Permutation and Embedding Transformation
  12. Framework Overview
  13. Perturbations of Adversarial Operations
  14. Framework and Defense Goals
  15. Permutation and Embedding Transformation based Randomized Smoothing
  16. ...and 37 more sections

Key Result

Theorem 1

Let $\phi_S: \mathcal{W} \times \mathbb{R}^n \to \mathcal{W}$ be the embedding substituting transformation based on a Staircase distribution $\varepsilon \sim \mathcal{S}_\gamma^\epsilon(w, \Delta)$ with PDF $f_\gamma^\epsilon(\cdot)$, and let $g_S$ be the smoothed classifier from any deterministic then $g_S(u\cdot \phi_S(w, \delta_{S}\cdot \Delta))=y_A$ for all $\|\delta_{S}\|_1 \leq \mathtt{RAD

Figures (14)

  • Figure 1: Text-CRS is a robustness certification framework based on randomized smoothing of permutation and embedding transformations against word-level adversarial attacks.
  • Figure 2: Word insertion and deletion. Blue and red indicate the changes to the permutation and embedding matrices.
  • Figure 3: An overview of Text-CRS.
  • Figure 4: PDF of synonym substitution for the word "good". The horizontal axis represents the embedding vector of each synonym. The vertical axis shows their probability.
  • Figure 5: The training toolkit can enhance model accuracy by replacing or supplementing part of the training process.
  • ...and 9 more figures

Theorems & Definitions (34)

  • Definition 1: $(\rho, \varepsilon)$-Smoothed Classifier
  • Definition 2: Staircase PDF geng2015staircase
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • Theorem 4
  • proof
  • Theorem 5
  • ...and 24 more