Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analyzing Multi-Head Attention on Trojan BERT Models

Jingwei Wang

TL;DR

The paper tackles the problem of NLP trojan attacks by examining how multi-head attention differs between trojan and benign BERT models in sentiment analysis. It introduces head-level categories—trigger heads, semantic heads, and specific heads—to explain trojan behavior and uses population-level statistics to show these patterns are widespread. Three attention-based detectors are proposed (naive, enumerate-trigger, and reverse-engineering) that leverage limited clean data to distinguish trojan from benign models, achieving near-perfect separation in at least one setting. The findings offer interpretability insights into Trojan NLP models and lay groundwork for practical detection and defense against backdoor attacks in language models.

Abstract

This project investigates the behavior of multi-head attention in Transformer models, specifically focusing on the differences between benign and trojan models in the context of sentiment analysis. Trojan attacks cause models to perform normally on clean inputs but exhibit misclassifications when presented with inputs containing predefined triggers. We characterize attention head functions in trojan and benign models, identifying specific 'trojan' heads and analyzing their behavior.

Analyzing Multi-Head Attention on Trojan BERT Models

TL;DR

The paper tackles the problem of NLP trojan attacks by examining how multi-head attention differs between trojan and benign BERT models in sentiment analysis. It introduces head-level categories—trigger heads, semantic heads, and specific heads—to explain trojan behavior and uses population-level statistics to show these patterns are widespread. Three attention-based detectors are proposed (naive, enumerate-trigger, and reverse-engineering) that leverage limited clean data to distinguish trojan from benign models, achieving near-perfect separation in at least one setting. The findings offer interpretability insights into Trojan NLP models and lay groundwork for practical detection and defense against backdoor attacks in language models.

Abstract

This project investigates the behavior of multi-head attention in Transformer models, specifically focusing on the differences between benign and trojan models in the context of sentiment analysis. Trojan attacks cause models to perform normally on clean inputs but exhibit misclassifications when presented with inputs containing predefined triggers. We characterize attention head functions in trojan and benign models, identifying specific 'trojan' heads and analyzing their behavior.

Paper Structure

This paper contains 20 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Threat Models
  3. Problem Definition
  4. Self-Generated Models
  5. Attention Pattern Exploration
  6. Distribution of Overall Heads Attention Weights
  7. Head-wise attention map
  8. Distribution of certain heads attention weights
  9. Head Functions
  10. Trigger Heads
  11. Semantic Heads
  12. Identify Semantic Heads
  13. Redirect Attention
  14. Redirect Importance to prediction
  15. Specific Heads
  16. ...and 5 more sections

Figures (7)

  • Figure 1: Trojan Attack Example, picture from wallace2019universal
  • Figure 2: Modern NLP model pipeline
  • Figure 3: Self-Generated Trojan and Benign models
  • Figure 4: Distribution of attention weights on trojan and benign models. Only keep those attention weights which are larger than 0.1.
  • Figure 5: Head-wise mean average max attention weights. Top subfigure is Trojan Model + Poisoned Input, bottom subfigure is Benign Model + Poisoned Input. x axis is layer number, y axis is head number.
  • ...and 2 more figures