Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-centric NLP Backdoor Defense from the Lens of Memorization

Zhenting Wang, Zhizhi Wang, Mingyu Jin, Mengnan Du, Juan Zhai, Shiqing Ma

TL;DR

This paper extends the definition of memorization of language models from sample-wise to more fine-grained sentence element-wise, and points out that language model backdoors are a type of element-wise memorization, and finds that the strength of such memorization is positively correlated to the frequency of duplicated elements in the training dataset.

Abstract

Backdoor attack is a severe threat to the trustworthiness of DNN-based language models. In this paper, we first extend the definition of memorization of language models from sample-wise to more fine-grained sentence element-wise (e.g., word, phrase, structure, and style), and then point out that language model backdoors are a type of element-wise memorization. Through further analysis, we find that the strength of such memorization is positively correlated to the frequency of duplicated elements in the training dataset. In conclusion, duplicated sentence elements are necessary for successful backdoor attacks. Based on this, we propose a data-centric defense. We first detect trigger candidates in training data by finding memorizable elements, i.e., duplicated elements, and then confirm real triggers by testing if the candidates can activate backdoor behaviors (i.e., malicious elements). Results show that our method outperforms state-of-the-art defenses in defending against different types of NLP backdoors.

Data-centric NLP Backdoor Defense from the Lens of Memorization

TL;DR

This paper extends the definition of memorization of language models from sample-wise to more fine-grained sentence element-wise, and points out that language model backdoors are a type of element-wise memorization, and finds that the strength of such memorization is positively correlated to the frequency of duplicated elements in the training dataset.

Abstract

Backdoor attack is a severe threat to the trustworthiness of DNN-based language models. In this paper, we first extend the definition of memorization of language models from sample-wise to more fine-grained sentence element-wise (e.g., word, phrase, structure, and style), and then point out that language model backdoors are a type of element-wise memorization. Through further analysis, we find that the strength of such memorization is positively correlated to the frequency of duplicated elements in the training dataset. In conclusion, duplicated sentence elements are necessary for successful backdoor attacks. Based on this, we propose a data-centric defense. We first detect trigger candidates in training data by finding memorizable elements, i.e., duplicated elements, and then confirm real triggers by testing if the candidates can activate backdoor behaviors (i.e., malicious elements). Results show that our method outperforms state-of-the-art defenses in defending against different types of NLP backdoors.
Paper Structure (27 sections, 2 theorems, 5 equations, 3 figures, 12 tables, 1 algorithm)

This paper contains 27 sections, 2 theorems, 5 equations, 3 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Memorization and Backdoor
  5. Backdoor as Memorization
  6. Method
  7. Overview
  8. STEP 1: Trigger Candidate Selection
  9. STEP 2: Trigger Verification
  10. Evaluation
  11. Setup
  12. Effectiveness
  13. Generalizability and Robustness
  14. Ablation Study
  15. Conclusion
  16. ...and 12 more sections

Key Result

Theorem 1

Given a model $\mathcal{M}$, the upper bound of the generalization error of memorization on backdoor trigger element $e_t$ is negatively correlated to the duplication frequency of $e_t$. Namely, when $Q(e_t)$ is higher, the upper bound of $\mathbb E \left(\ell(\mathcal{M}(x),y_t)\right)$ (where $e_t

Figures (3)

  • Figure 1: Sample-wise and element-wise memorization.
  • Figure 2: An example of the instantialized syntax tree with elements at different levels.
  • Figure 3: Duplication on trigger element makes backdoor related memorization.

Theorems & Definitions (6)

  • Definition 1
  • Definition 2
  • Definition 3
  • Theorem 1
  • Lemma 1
  • proof