Thieves on Sesame Street! Model Extraction of BERT-based APIs

TL;DR

The paper demonstrates that NLP APIs built on fine-tuned BERT representations are highly vulnerable to query-based model extraction, even when adversaries have no access to training data. By using random or nonsensical inputs, paired with task-specific heuristics, an attacker can train a local copy that attains near-victim performance across SST2, MNLI, SQuAD, and BoolQ at modest query costs, with pretraining on the attacker side further boosting success. The authors also evaluate defenses—membership classification and watermarking—and show that they offer only partial resilience against adaptive adversaries. Overall, the work highlights a substantial security and IP risk introduced by transfer learning in NLP and motivates the development of stronger, adversary-aware defenses and defense-aware evaluation methodologies.

Abstract

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al. 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction---membership classification and API watermarking---which while successful against naive adversaries, are ineffective against more sophisticated ones.

Figures (3)

• Figure 1: Overview of our model extraction setup for question answering.An attacker first queries a victim BERT model, and then uses its predicted answers to fine-tune their own BERT model. This process works even when passages and questions are random sequences of words as shown here.
• Figure 2: Average dev F1 for extracted SQuAD models after selecting different subsets of data from a large pool of wiki and random data. Subsets are selected based on the agreement between the outputs of different runs of the original SQuAD model. Notice the large difference between the highest agreement (blue) and the lowest agreement (green), especially at small dataset sizes.
• Figure 3: Histogram of average F1 agreement between five different runs of BERT question answering models trained on the original SQuAD dataset. Notice the higher agreement on points in the wiki dataset compared to random.