Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Certifying Language Model Robustness with Fuzzed Randomized Smoothing: An Efficient Defense Against Backdoor Attacks

Bowei He, Lihao Yin, Hui-Ling Zhen, Jianping Zhang, Lanqing Hong, Mingxuan Yuan, Chen Ma

TL;DR

This work tackles the challenge of protecting pre-trained language models from textual backdoor attacks planted during pre-training by introducing Fuzzed Randomized Smoothing (FRS). The approach seamlessly combines Monte Carlo tree search–driven fuzzing to locate vulnerable text segments with biphased model parameter smoothing and randomized smoothing to certify robustness without poisoned training data. Theoretical guarantees extend robustness radii beyond existing methods, and extensive experiments across multiple models, datasets, and attack strategies show superior defense performance and broader certified robustness. Practically, FRS offers a scalable, data-efficient defense mechanism that enhances reliability for high-stakes NLP applications while maintaining generation and downstream-task quality.

Abstract

The widespread deployment of pre-trained language models (PLMs) has exposed them to textual backdoor attacks, particularly those planted during the pre-training stage. These attacks pose significant risks to high-reliability applications, as they can stealthily affect multiple downstream tasks. While certifying robustness against such threats is crucial, existing defenses struggle with the high-dimensional, interdependent nature of textual data and the lack of access to original poisoned pre-training data. To address these challenges, we introduce \textbf{F}uzzed \textbf{R}andomized \textbf{S}moothing (\textbf{FRS}), a novel approach for efficiently certifying language model robustness against backdoor attacks. FRS integrates software robustness certification techniques with biphased model parameter smoothing, employing Monte Carlo tree search for proactive fuzzing to identify vulnerable textual segments within the Damerau-Levenshtein space. This allows for targeted and efficient text randomization, while eliminating the need for access to poisoned training data during model smoothing. Our theoretical analysis demonstrates that FRS achieves a broader certified robustness radius compared to existing methods. Extensive experiments across various datasets, model configurations, and attack strategies validate FRS's superiority in terms of defense efficiency, accuracy, and robustness.

Certifying Language Model Robustness with Fuzzed Randomized Smoothing: An Efficient Defense Against Backdoor Attacks

TL;DR

This work tackles the challenge of protecting pre-trained language models from textual backdoor attacks planted during pre-training by introducing Fuzzed Randomized Smoothing (FRS). The approach seamlessly combines Monte Carlo tree search–driven fuzzing to locate vulnerable text segments with biphased model parameter smoothing and randomized smoothing to certify robustness without poisoned training data. Theoretical guarantees extend robustness radii beyond existing methods, and extensive experiments across multiple models, datasets, and attack strategies show superior defense performance and broader certified robustness. Practically, FRS offers a scalable, data-efficient defense mechanism that enhances reliability for high-stakes NLP applications while maintaining generation and downstream-task quality.

Abstract

The widespread deployment of pre-trained language models (PLMs) has exposed them to textual backdoor attacks, particularly those planted during the pre-training stage. These attacks pose significant risks to high-reliability applications, as they can stealthily affect multiple downstream tasks. While certifying robustness against such threats is crucial, existing defenses struggle with the high-dimensional, interdependent nature of textual data and the lack of access to original poisoned pre-training data. To address these challenges, we introduce \textbf{F}uzzed \textbf{R}andomized \textbf{S}moothing (\textbf{FRS}), a novel approach for efficiently certifying language model robustness against backdoor attacks. FRS integrates software robustness certification techniques with biphased model parameter smoothing, employing Monte Carlo tree search for proactive fuzzing to identify vulnerable textual segments within the Damerau-Levenshtein space. This allows for targeted and efficient text randomization, while eliminating the need for access to poisoned training data during model smoothing. Our theoretical analysis demonstrates that FRS achieves a broader certified robustness radius compared to existing methods. Extensive experiments across various datasets, model configurations, and attack strategies validate FRS's superiority in terms of defense efficiency, accuracy, and robustness.

Paper Structure

This paper contains 76 sections, 4 theorems, 39 equations, 1 figure, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Methodology
  5. Randomized Smoothing Defense
  6. Biphased Model Parameter Smoothing
  7. Fuzzed Text Randomization
  8. Vulnerable Area Identification
  9. Text Randomization
  10. Theoretical Robustness Bound
  11. Experiments
  12. Experiment Setup
  13. Experiment Results and Analysis
  14. Defense Performance (RQ1)
  15. Certified Robustness Radius (RQ2)
  16. ...and 61 more sections

Key Result

Theorem 1

Based on the Assumption assump: effective parameter smoothing, the lower bound of the probability that the smoothed model $\tilde{f}$ returns the $y*$ for perturbed input $\mathbf{x}'$ after the randomized smoothing $\underline{p_{y*}(\mathbf{x}')} = Beta(\alpha; K_{y*}, K - K_{y*} + 1)$, where $Bet , with probability at least $1-\alpha$: $\tilde{f}(\mathbf{x}') = y*$. Here, $\Delta$ denotes the p

Figures (1)

  • Figure 1: The comparison between our FRS method and no defense method on ASR metric in different datasets. The BadPre is taken as the pre-training attack method. 'b' and 'l' are short for 'base' and 'large', respectively.

Theorems & Definitions (6)

  • Theorem 1: Model Robustness Condition
  • Corollary 1: Broader Robustness Radius
  • Theorem 2
  • proof
  • Theorem 3
  • proof